Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 1481699 Details for
Bug 1624527
MDS spams is_laggy message at log level 1
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
Hotfix_1624527_report
Hotfix_1624527_report (text/plain), 24.19 KB, created by
Ramakrishnan Periyasamy
on 2018-09-08 05:41:59 UTC
(
hide
)
Description:
Hotfix_1624527_report
Filename:
MIME Type:
Creator:
Ramakrishnan Periyasamy
Created:
2018-09-08 05:41:59 UTC
Size:
24.19 KB
patch
obsolete
>Hotfix_1624527 Report: >------------------------- > >Configured cluster with 3.0z5 GA based build and able to reproduce the issue. Console output and logs added below. > >Note: Client IO was running continuously during this entire test execution >IO: for i in {1..100};do sudo cp -av /usr /mnt/fuse/fuse/dir4/.; done > >Cluster verison before upgrade: >------------------------------- >[ubuntu@magna051 ~]$ ceph -v >ceph version 12.2.4-42.el7cp (f73642baacccbf2a3c254d1fb5f0317b933b28cf) luminous (stable) >[ubuntu@magna051 ~]$ >[ubuntu@magna051 ~]$ sudo ceph -s > cluster: > id: 366f0928-0367-4fb0-ab6b-939700f59f40 > health: HEALTH_OK > > services: > mon: 1 daemons, quorum magna051 > mgr: magna051(active) > mds: cephfs-2/2/2 up {0=magna092=up:active,1=magna113=up:active}, 1 up:standby > osd: 9 osds: 9 up, 9 in > > data: > pools: 2 pools, 128 pgs > objects: 244k objects, 37363 MB > usage: 114 GB used, 8220 GB / 8334 GB avail > pgs: 128 active+clean > > io: > client: 683 kB/s wr, 0 op/s rd, 162 op/s wr > >[ubuntu@magna092 ~]$ sudo ss -tnp | grep ceph-mds >ESTAB 0 0 10.8.128.92:60148 10.8.128.68:6808 users:(("ceph-mds",pid=3430,fd=31)) >ESTAB 0 0 10.8.128.92:32838 10.8.128.56:6808 users:(("ceph-mds",pid=3430,fd=30)) >ESTAB 0 0 10.8.128.92:44122 10.8.128.51:6789 users:(("ceph-mds",pid=3430,fd=17)) >ESTAB 0 0 10.8.128.92:47418 10.8.128.56:6800 users:(("ceph-mds",pid=3430,fd=34)) >ESTAB 0 0 10.8.128.92:37116 10.8.128.68:6804 users:(("ceph-mds",pid=3430,fd=28)) >ESTAB 0 0 10.8.128.92:56194 10.8.128.51:6800 users:(("ceph-mds",pid=3430,fd=26)) >ESTAB 0 0 10.8.128.92:6800 10.8.128.70:43614 users:(("ceph-mds",pid=3430,fd=37)) >ESTAB 0 0 10.8.128.92:47384 10.8.128.52:6808 users:(("ceph-mds",pid=3430,fd=35)) >ESTAB 0 0 10.8.128.92:53148 10.8.128.56:6804 users:(("ceph-mds",pid=3430,fd=29)) >ESTAB 0 0 10.8.128.92:55922 10.8.128.68:6800 users:(("ceph-mds",pid=3430,fd=27)) >ESTAB 0 0 10.8.128.92:60522 10.8.128.52:6804 users:(("ceph-mds",pid=3430,fd=32)) >ESTAB 0 0 10.8.128.92:48830 10.8.128.52:6800 users:(("ceph-mds",pid=3430,fd=33)) >(reverse-i-search)`d': sudo ss -tnp | grep ceph-m^C >[ubuntu@magna092 ~]$ sudo iptables -I INPUT -p tcp --sport 44122 --dport 6789 -j DROP > >Mds log: >---------------- >2018-09-08 04:47:21.867619 7fbe2e5a8700 10 mds.beacon.magna092 _send up:active seq 15782 >2018-09-08 04:47:25.867723 7fbe2e5a8700 10 mds.beacon.magna092 _send up:active seq 15783 >2018-09-08 04:47:28.867744 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.000240 > 15 since last acked beacon >2018-09-08 04:47:28.869684 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.002181 > 15 since last acked beacon >2018-09-08 04:47:28.869698 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.002194 > 15 since last acked beacon >2018-09-08 04:47:28.869700 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.002197 > 15 since last acked beacon >2018-09-08 04:47:29.280596 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413092 > 15 since last acked beacon >2018-09-08 04:47:29.280619 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413115 > 15 since last acked beacon >2018-09-08 04:47:29.280622 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413118 > 15 since last acked beacon >2018-09-08 04:47:29.280640 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413136 > 15 since last acked beacon >2018-09-08 04:47:29.280649 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413145 > 15 since last acked beacon >2018-09-08 04:47:29.280651 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413147 > 15 since last acked beacon >2018-09-08 04:47:29.280665 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413161 > 15 since last acked beacon >2018-09-08 04:47:29.280674 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413170 > 15 since last acked beacon >2018-09-08 04:47:29.280676 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413173 > 15 since last acked beacon >2018-09-08 04:47:29.280690 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413186 > 15 since last acked beacon >2018-09-08 04:47:29.280698 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413194 > 15 since last acked beacon >2018-09-08 04:47:29.280701 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413197 > 15 since last acked beacon >2018-09-08 04:47:29.280732 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413228 > 15 since last acked beacon >2018-09-08 04:47:29.280747 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413243 > 15 since last acked beacon >2018-09-08 04:47:29.280750 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413246 > 15 since last acked beacon >2018-09-08 04:47:29.280808 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413304 > 15 since last acked beacon >2018-09-08 04:47:29.280819 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413316 > 15 since last acked beacon >2018-09-08 04:47:29.280822 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413318 > 15 since last acked beacon >2018-09-08 04:47:29.280905 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413401 > 15 since last acked beacon >2018-09-08 04:47:29.280917 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413413 > 15 since last acked beacon >2018-09-08 04:47:29.280919 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413416 > 15 since last acked beacon >2018-09-08 04:47:29.280991 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413487 > 15 since last acked beacon >2018-09-08 04:47:29.281008 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413504 > 15 since last acked beacon >2018-09-08 04:47:29.281012 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413508 > 15 since last acked beacon >2018-09-08 04:47:29.281069 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413565 > 15 since last acked beacon >2018-09-08 04:47:29.281085 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413581 > 15 since last acked beacon >2018-09-08 04:47:29.281088 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413584 > 15 since last acked beacon >2018-09-08 04:47:29.281103 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413599 > 15 since last acked beacon >2018-09-08 04:47:29.281112 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413608 > 15 since last acked beacon >2018-09-08 04:47:29.281115 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413611 > 15 since last acked beacon >2018-09-08 04:47:29.281161 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413657 > 15 since last acked beacon >2018-09-08 04:47:29.281175 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413671 > 15 since last acked beacon >2018-09-08 04:47:29.281178 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413674 > 15 since last acked beacon >2018-09-08 04:47:29.281214 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413711 > 15 since last acked beacon >2018-09-08 04:47:29.281226 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413722 > 15 since last acked beacon >2018-09-08 04:47:29.281229 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413725 > 15 since last acked beacon >2018-09-08 04:47:29.281290 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413786 > 15 since last acked beacon >2018-09-08 04:47:29.281304 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413800 > 15 since last acked beacon >2018-09-08 04:47:29.281306 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413803 > 15 since last acked beacon >2018-09-08 04:47:29.281344 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413840 > 15 since last acked beacon >2018-09-08 04:47:29.281356 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413852 > 15 since last acked beacon >2018-09-08 04:47:29.281359 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413855 > 15 since last acked beacon >2018-09-08 04:47:29.281407 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413903 > 15 since last acked beacon >2018-09-08 04:47:29.281418 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413914 > 15 since last acked beacon >2018-09-08 04:47:29.281421 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413917 > 15 since last acked beacon >2018-09-08 04:47:29.867832 7fbe2e5a8700 10 mds.beacon.magna092 _send up:active seq 15784 >2018-09-08 04:47:30.280741 7fbe315ae700 5 mds.beacon.magna092 is_laggy 16.413237 > 15 since last acked beacon >2018-09-08 04:47:30.280759 7fbe315ae700 5 mds.beacon.magna092 is_laggy 16.413255 > 15 since last acked beacon >2018-09-08 04:47:30.280762 7fbe315ae700 5 mds.beacon.magna092 is_laggy 16.413259 > 15 since last acked beacon >2018-09-08 04:47:30.280781 7fbe315ae700 5 mds.beacon.magna092 is_laggy 16.413277 > 15 since last acked beacon >2018-09-08 04:47:30.280792 7fbe315ae700 5 mds.beacon.magna092 is_laggy 16.413289 > 15 since last acked beacon >2018-09-08 04:47:30.280796 7fbe315ae700 5 mds.beacon.magna092 is_laggy 16.413292 > 15 since last acked beacon >2018-09-08 04:47:30.280813 7fbe315ae700 5 mds.beacon.magna092 is_laggy 16.413309 > 15 since last acked beacon >2018-09-08 04:47:30.280824 7fbe315ae700 5 mds.beacon.magna092 is_laggy 16.413320 > 15 since last acked beacon >2018-09-08 04:47:30.280827 7fbe315ae700 5 mds.beacon.magna092 is_laggy 16.413323 > 15 since last acked beacon >2018-09-08 04:47:30.280839 7fbe315ae700 5 mds.beacon.magna092 is_laggy 16.413335 > 15 since last acked beacon >2018-09-08 04:47:30.280847 7fbe315ae700 5 mds.beacon.magna092 is_laggy 16.413343 > 15 since last acked beacon >2018-09-08 04:47:30.280850 7fbe315ae700 5 mds.beacon.magna092 is_laggy 16.413346 > 15 since last acked beacon >2018-09-08 04:47:30.280862 7fbe315ae700 5 mds.beacon.magna092 is_laggy 16.413358 > 15 since last acked beacon >2018-09-08 04:47:30.280870 7fbe315ae700 5 mds.beacon.magna092 is_laggy 16.413366 > 15 since last acked beacon >2018-09-08 04:47:30.280872 7fbe315ae700 5 mds.beacon.magna092 is_laggy 16.413369 > 15 since last acked beacon >2018-09-08 04:47:30.280885 7fbe315ae700 5 mds.beacon.magna092 is_laggy 16.413382 > 15 since last acked beacon >2018-09-08 04:47:30.280894 7fbe315ae700 5 mds.beacon.magna092 is_laggy 16.413390 > 15 since last acked beacon >2018-09-08 04:47:30.280896 7fbe315ae700 5 mds.beacon.magna092 is_laggy 16.413393 > 15 since last acked beacon >2018-09-08 04:47:30.280954 7fbe315ae700 5 mds.beacon.magna092 is_laggy 16.413451 > 15 since last acked beacon >2018-09-08 04:47:30.280964 7fbe315ae700 5 mds.beacon.magna092 is_laggy 16.413460 > 15 since last acked beacon >2018-09-08 04:47:30.280966 7fbe315ae700 5 mds.beacon.magna092 is_laggy 16.413462 > 15 since last acked beacon > -163> 2018-09-08 04:47:28.867744 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.000240 > 15 since last acked beacon > -160> 2018-09-08 04:47:28.869684 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.002181 > 15 since last acked beacon > -158> 2018-09-08 04:47:28.869698 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.002194 > 15 since last acked beacon > -157> 2018-09-08 04:47:28.869700 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.002197 > 15 since last acked beacon > -153> 2018-09-08 04:47:29.280596 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413092 > 15 since last acked beacon > -151> 2018-09-08 04:47:29.280619 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413115 > 15 since last acked beacon > -150> 2018-09-08 04:47:29.280622 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413118 > 15 since last acked beacon > -147> 2018-09-08 04:47:29.280640 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413136 > 15 since last acked beacon > -145> 2018-09-08 04:47:29.280649 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413145 > 15 since last acked beacon > -144> 2018-09-08 04:47:29.280651 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413147 > 15 since last acked beacon > -142> 2018-09-08 04:47:29.280665 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413161 > 15 since last acked beacon > -139> 2018-09-08 04:47:29.280674 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413170 > 15 since last acked beacon > -138> 2018-09-08 04:47:29.280676 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413173 > 15 since last acked beacon > -136> 2018-09-08 04:47:29.280690 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413186 > 15 since last acked beacon > -134> 2018-09-08 04:47:29.280698 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413194 > 15 since last acked beacon > -133> 2018-09-08 04:47:29.280701 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413197 > 15 since last acked beacon > -130> 2018-09-08 04:47:29.280732 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413228 > 15 since last acked beacon > -128> 2018-09-08 04:47:29.280747 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413243 > 15 since last acked beacon > -127> 2018-09-08 04:47:29.280750 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413246 > 15 since last acked beacon > -124> 2018-09-08 04:47:29.280808 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413304 > 15 since last acked beacon > -122> 2018-09-08 04:47:29.280819 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413316 > 15 since last acked beacon > -121> 2018-09-08 04:47:29.280822 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413318 > 15 since last acked beacon > -118> 2018-09-08 04:47:29.280905 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413401 > 15 since last acked beacon > -116> 2018-09-08 04:47:29.280917 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413413 > 15 since last acked beacon > -115> 2018-09-08 04:47:29.280919 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413416 > 15 since last acked beacon > -112> 2018-09-08 04:47:29.280991 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413487 > 15 since last acked beacon > -110> 2018-09-08 04:47:29.281008 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413504 > 15 since last acked beacon > -109> 2018-09-08 04:47:29.281012 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413508 > 15 since last acked beacon > -105> 2018-09-08 04:47:29.281069 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413565 > 15 since last acked beacon > -103> 2018-09-08 04:47:29.281085 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413581 > 15 since last acked beacon > -102> 2018-09-08 04:47:29.281088 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413584 > 15 since last acked beacon > -100> 2018-09-08 04:47:29.281103 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413599 > 15 since last acked beacon > -98> 2018-09-08 04:47:29.281112 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413608 > 15 since last acked beacon > -97> 2018-09-08 04:47:29.281115 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413611 > 15 since last acked beacon > -94> 2018-09-08 04:47:29.281161 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413657 > 15 since last acked beacon > -92> 2018-09-08 04:47:29.281175 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413671 > 15 since last acked beacon > -91> 2018-09-08 04:47:29.281178 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413674 > 15 since last acked beacon > -88> 2018-09-08 04:47:29.281214 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413711 > 15 since last acked beacon > -86> 2018-09-08 04:47:29.281226 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413722 > 15 since last acked beacon > -85> 2018-09-08 04:47:29.281229 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413725 > 15 since last acked beacon > -82> 2018-09-08 04:47:29.281290 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413786 > 15 since last acked beacon > -80> 2018-09-08 04:47:29.281304 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413800 > 15 since last acked beacon > -79> 2018-09-08 04:47:29.281306 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413803 > 15 since last acked beacon > -76> 2018-09-08 04:47:29.281344 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413840 > 15 since last acked beacon > -74> 2018-09-08 04:47:29.281356 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413852 > 15 since last acked beacon > -73> 2018-09-08 04:47:29.281359 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413855 > 15 since last acked beacon > -70> 2018-09-08 04:47:29.281407 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413903 > 15 since last acked beacon > -68> 2018-09-08 04:47:29.281418 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413914 > 15 since last acked beacon > -67> 2018-09-08 04:47:29.281421 7fbe315ae700 5 mds.beacon.magna092 is_laggy 15.413917 > 15 since last acked beacon > -66> 2018-09-08 04:47:29.867832 7fbe2e5a8700 10 mds.beacon.magna092 _send up:active seq 15784 > -64> 2018-09-08 04:47:29.867853 7fbe2e5a8700 1 -- 10.8.128.92:6800/4096239442 --> 10.8.128.51:6789/0 -- mdsbeacon(14183/magna092 up:active seq 15784 v194) v7 -- 0x558cf182e000 con 0 > > >Cluster Upgrade: >----------------------- >Upgraded Cluster to hotfix build with IO from client. >Followed MDS upgrade procedures https://github.com/ceph/ceph/blob/luminous/doc/cephfs/upgrading.rst > >After upgrade >------------------------ >[ubuntu@magna051 ~]$ ceph -v >ceph version 12.2.4-42.1.hotfix.nvidia.el7cp (4a72ecd06cdc5a049945b166073ce39fbe631308) luminous (stable) >[ubuntu@magna051 ~]$ sudo ceph -s > cluster: > id: 366f0928-0367-4fb0-ab6b-939700f59f40 > health: HEALTH_OK > > services: > mon: 1 daemons, quorum magna051 > mgr: magna051(active) > mds: cephfs-2/2/2 up {0=magna113=up:active,1=magna092=up:active}, 1 up:standby > osd: 9 osds: 9 up, 9 in > > data: > pools: 2 pools, 128 pgs > objects: 249k objects, 37274 MB > usage: 113 GB used, 8220 GB / 8334 GB avail > pgs: 128 active+clean > > io: > client: 511 B/s rd, 209 kB/s wr, 0 op/s rd, 76 op/s wr > >[ubuntu@magna051 ~]$ sudo ceph fs status >cephfs - 1 clients >====== >+------+--------+----------+---------------+-------+-------+ >| Rank | State | MDS | Activity | dns | inos | >+------+--------+----------+---------------+-------+-------+ >| 0 | active | magna113 | Reqs: 0 /s | 36.9k | 32.8k | >| 1 | active | magna114 | Reqs: 0 /s | 33.2k | 29.4k | >+------+--------+----------+---------------+-------+-------+ >+-----------------+----------+-------+-------+ >| Pool | type | used | avail | >+-----------------+----------+-------+-------+ >| cephfs_metadata | metadata | 377M | 2591G | >| cephfs_data | data | 36.0G | 2591G | >+-----------------+----------+-------+-------+ > >+-------------+ >| Standby MDS | >+-------------+ >| magna092 | >+-------------+ >MDS version: ceph version 12.2.4-42.1.hotfix.nvidia.el7cp (4a72ecd06cdc5a049945b166073ce39fbe631308) luminous (stable) > >[ubuntu@magna092 ~]$ sudo ss -tnp | grep ceph-mds >ESTAB 0 0 10.8.128.92:60256 10.8.128.68:6808 users:(("ceph-mds",pid=6214,fd=26)) >ESTAB 0 0 10.8.128.92:56036 10.8.128.68:6800 users:(("ceph-mds",pid=6214,fd=27)) >ESTAB 0 0 10.8.128.92:37242 10.8.128.68:6804 users:(("ceph-mds",pid=6214,fd=34)) >ESTAB 0 0 10.8.128.92:32972 10.8.128.56:6808 users:(("ceph-mds",pid=6214,fd=33)) >ESTAB 0 0 10.8.128.92:48956 10.8.128.52:6800 users:(("ceph-mds",pid=6214,fd=30)) >ESTAB 0 0 10.8.128.92:44240 10.8.128.51:6789 users:(("ceph-mds",pid=6214,fd=17)) >ESTAB 0 0 10.8.128.92:47544 10.8.128.56:6800 users:(("ceph-mds",pid=6214,fd=31)) >ESTAB 0 0 10.8.128.92:53276 10.8.128.56:6804 users:(("ceph-mds",pid=6214,fd=29)) >ESTAB 0 0 10.8.128.92:56338 10.8.128.51:6800 users:(("ceph-mds",pid=6214,fd=35)) >ESTAB 0 0 10.8.128.92:6800 10.8.128.114:35452 users:(("ceph-mds",pid=6214,fd=38)) >ESTAB 0 0 10.8.128.92:60650 10.8.128.52:6804 users:(("ceph-mds",pid=6214,fd=28)) >ESTAB 0 0 10.8.128.92:47510 10.8.128.52:6808 users:(("ceph-mds",pid=6214,fd=32)) >ESTAB 0 0 10.8.128.92:6800 10.8.128.70:44872 users:(("ceph-mds",pid=6214,fd=36)) >[ubuntu@magna092 ~]$ sudo iptables -I OUTPUT -p tcp --sport 44240 --dport 6789 -j DROP > >MDS logs: >------------------------ >2018-09-08 05:24:52.828060 7f8564705700 5 mds.beacon.magna092 handle_mds_beacon up:active seq 200 rtt 0.000579 >2018-09-08 05:24:56.827548 7f855ef7f700 5 mds.beacon.magna092 _send up:active seq 201 >2018-09-08 05:25:00.827659 7f855ef7f700 5 mds.beacon.magna092 _send up:active seq 202 >2018-09-08 05:25:04.827741 7f855ef7f700 5 mds.beacon.magna092 _send up:active seq 203 >2018-09-08 05:25:07.851715 7f8561f85700 1 mds.beacon.magna092 MDS connection to Monitors appears to be laggy; 15.024234s since last acked beacon >2018-09-08 05:25:08.827859 7f855ef7f700 5 mds.beacon.magna092 _send up:active seq 204 > -9265> 2018-09-08 05:24:48.827390 7f855ef7f700 5 mds.beacon.magna092 _send up:active seq 199 > -9263> 2018-09-08 05:24:48.827409 7f855ef7f700 1 -- 10.8.128.92:6800/3178541370 --> 10.8.128.51:6789/0 -- mdsbeacon(24206/magna092 up:active seq 199 v239) v7 -- 0x557acb654380 con 0 > -9262> 2018-09-08 05:24:48.827800 7f8564705700 5 -- 10.8.128.92:6800/3178541370 >> 10.8.128.51:6789/0 conn(0x557ac9445000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=124 cs=1 l=1). rx mon.0 seq 217 0x557acb654380 mdsbeacon(24206/magna092 up:active seq 199 v239) v7 > -9261> 2018-09-08 05:24:48.827818 7f8564705700 1 -- 10.8.128.92:6800/3178541370 <== mon.0 10.8.128.51:6789/0 217 ==== mdsbeacon(24206/magna092 up:active seq 199 v239) v7 ==== 133+0+0 (3166973937 0 0) 0x557acb654380 con 0x557ac9445000 > -9260> 2018-09-08 05:24:48.827827 7f8564705700 5 mds.beacon.magna092 handle_mds_beacon up:active seq 199 rtt 0.000428 > -7056> 2018-09-08 05:24:52.827470 7f855ef7f700 5 mds.beacon.magna092 _send up:active seq 200 > -7054> 2018-09-08 05:24:52.827490 7f855ef7f700 1 -- 10.8.128.92:6800/3178541370 --> 10.8.128.51:6789/0 -- mdsbeacon(24206/magna092 up:active seq 200 v239) v7 -- 0x557acb655180 con 0 > -7053> 2018-09-08 05:24:52.828035 7f8564705700 5 -- 10.8.128.92:6800/3178541370 >> 10.8.128.51:6789/0 conn(0x557ac9445000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=124 cs=1 l=1). rx mon.0 seq 218 0x557acb655180 mdsbeacon(24206/magna092 up:active seq 200 v239) v7 > -7052> 2018-09-08 05:24:52.828052 7f8564705700 1 -- 10.8.128.92:6800/3178541370 <== mon.0 10.8.128.51:6789/0 218 ==== mdsbeacon(24206/magna092 up:active seq 200 v239) v7 ==== 133+0+0 (4218276460 0 0) 0x557acb655180 con 0x557ac9445000 > -7051> 2018-09-08 05:24:52.828060 7f8564705700 5 mds.beacon.magna092 handle_mds_beacon up:active seq 200 rtt 0.000579 > -6856> 2018-09-08 05:24:56.827548 7f855ef7f700 5 mds.beacon.magna092 _send up:active seq 201 > -6854> 2018-09-08 05:24:56.827571 7f855ef7f700 1 -- 10.8.128.92:6800/3178541370 --> 10.8.128.51:6789/0 -- mdsbeacon(24206/magna092 up:active seq 201 v239) v7 -- 0x557acb655500 con 0 > -3585> 2018-09-08 05:25:00.827659 7f855ef7f700 5 mds.beacon.magna092 _send up:active seq 202 > -3583> 2018-09-08 05:25:00.827677 7f855ef7f700 1 -- 10.8.128.92:6800/3178541370 --> 10.8.128.51:6789/0 -- mdsbeacon(24206/magna092 up:active seq 202 v239) v7 -- 0x557ad1bd0380 con 0 > -2129> 2018-09-08 05:25:04.827741 7f855ef7f700 5 mds.beacon.magna092 _send up:active seq 203 > -2127> 2018-09-08 05:25:04.827760 7f855ef7f700 1 -- 10.8.128.92:6800/3178541370 --> 10.8.128.51:6789/0 -- mdsbeacon(24206/magna092 up:active seq 203 v239) v7 -- 0x557ad1bd0700 con 0 > -32> 2018-09-08 05:25:07.851715 7f8561f85700 1 mds.beacon.magna092 MDS connection to Monitors appears to be laggy; 15.024234s since last acked beacon > -30> 2018-09-08 05:25:08.827859 7f855ef7f700 5 mds.beacon.magna092 _send up:active seq 204 > -28> 2018-09-08 05:25:08.827878 7f855ef7f700 1 -- 10.8.128.92:6800/3178541370 --> 10.8.128.51:6789/0 -- mdsbeacon(24206/magna092 up:active seq 204 v239) v7 -- 0x557ad1bd1180 con 0 >2018-09-08 05:25:11.143245 7f0591e38240 5 mds.beacon.magna092 _send up:boot seq 1 >2018-09-08 05:25:15.143436 7f0587f6a700 5 mds.beacon.magna092 _send up:boot seq 2 >2018-09-08 05:25:15.553912 7f058af70700 5 mds.beacon.magna092 set_want_state: up:boot -> up:standby >2018-09-08 05:25:15.554028 7f058d6f0700 5 mds.beacon.magna092 handle_mds_beacon up:boot seq 2 rtt 0.410583 >2018-09-08 05:25:19.143638 7f0587f6a700 5 mds.beacon.magna092 _send up:standby seq 3 >2018-09-08 05:25:19.144232 7f058d6f0700 5 mds.beacon.magna092 handle_mds_beacon up:standby seq 3 rtt 0.000582 > >
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 1624527
: 1481699