Bug 1470828

Summary: ceph osd pool create manila_metadata 32 returned 22 instead of one of [0]
Product: Red Hat OpenStack Reporter: Gonéri Le Bouder <goneri>
Component: puppet-cephAssignee: Sébastien Han <shan>
Status: CLOSED WONTFIX QA Contact: Yogev Rabl <yrabl>
Severity: medium Docs Contact:
Priority: medium    
Version: 11.0 (Ocata)CC: aschultz, gfidente, goneri, icolle, jjoyce, jomurphy, jschluet, seb, slinaber, tvignaud
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: 11.0 (Ocata)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-09-25 20:56:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
puppet.log on controller0 none

Description Gonéri Le Bouder 2017-07-13 18:42:49 UTC
Created attachment 1297801 [details]
puppet.log on controller0

Description of problem:

I got this error during an OSP11 deployment with the last puddle:

Error: /bin/true # comment to satisfy puppet syntax requirements
set -ex
ceph osd pool create manila_metadata 32 returned 22 instead of one of [0]
Error: /Stage[main]/Tripleo::Profile::Base::Ceph::Mon/Ceph::Pool[manila_metadata]/Exec[create-manila_metadata]/returns: change from notrun to 0 failed: /bin/true # comment to satisfy puppet syntax requirements
set -ex
ceph osd pool create manila_metadata 32 returned 22 instead of one of [0]

I can run the command manually without any issue:

[heat-admin@overcloud-controller-0 ~]$ sudo ceph osd pool create manila_metadata 32
pool 'manila_metadata' already exists
[heat-admin@overcloud-controller-0 ~]$ echo $?
0

Comment 1 Gonéri Le Bouder 2017-07-13 18:46:33 UTC
Underlying error is: "crushtool check failed with -22"

Notice: /Stage[main]/Tripleo::Profile::Base::Ceph::Mon/Ceph::Pool[manila_metadata]/Exec[create-manila_metadata]/returns: + ceph osd pool create manila_metadata 32
Notice: /Stage[main]/Tripleo::Profile::Base::Ceph::Mon/Ceph::Pool[manila_metadata]/Exec[create-manila_metadata]/returns: Error EINVAL: crushtool check failed with -22: global/signal_handler.cc: In function 'void SignalHandler::queue_signal(int)' thread 7f223d5ec700 time 2017-07-13 18:11:20.249208
Notice: /Stage[main]/Tripleo::Profile::Base::Ceph::Mon/Ceph::Pool[manila_metadata]/Exec[set-manila_metadata-pg_num]: Dependency Exec[create-manila_metadata] has failures: true
Notice: /Stage[main]/Tripleo::Profile::Base::Ceph::Mon/Ceph::Pool[manila_metadata]/Exec[set-manila_metadata-pgp_num]: Dependency Exec[create-manila_metadata] has failures: true
Notice: /Stage[main]/Tripleo::Profile::Base::Ceph::Mon/Ceph::Pool[manila_metadata]/Exec[set-manila_metadata-size]: Dependency Exec[create-manila_metadata] has failures: true

Comment 2 seb 2017-08-02 16:38:30 UTC
Looks like a Ceph bug that should be investigated by the Ceph core team.
The Ceph DFG can not do much.

Comment 3 seb 2017-08-03 15:29:48 UTC
Just to clarify my last comment, I should have been clearer.
I don't believe it's a bug in Ceph core, I'm saying a Ceph core should check this error.
I believe this might be an ordering/sequencing issue, if a dev can come up with something clearer regarding this error then we can determine which part of the sequence is causing this :).

Comment 7 Giulio Fidente 2017-09-20 14:40:03 UTC
Can you attach Ceph logs from /var/log/ceph from all controller nodes?

Comment 9 Gonéri Le Bouder 2017-09-20 15:56:47 UTC
Sorry Giulio but I've nothing more than what I attached to the bug.