Description of problem:
Previously, Crimson OSD deployments through cephadm required the users to set these 3 vital config options before triggering OSD deployment:
    - ceph config set global 'enable_experimental_unrecoverable_data_corrupting_features' crimson
    - ceph osd set-allow-crimson --yes-i-really-mean-it
    - ceph config set mon osd_pool_default_crimson true
  Refer: 
    - https://docs.ceph.com/en/latest/dev/crimson/crimson/?highlight=crimson#enabling-crimson
    - https://docs.redhat.com/en/documentation/red_hat_ceph_storage/7/html/administration_guide/crimson#configuring-crimson_admin
    - https://www.ibm.com/docs/en/storage-ceph/7.1?topic=preview-configuring-crimson

However, with recent changes introduced around Fed 2024 with PR - https://github.com/ceph/ceph/pull/55276, the user was now required to also assign CPU core per OSD basis for optimal deployment and better performance. Notably the parameters in this PR were supported in a VStart deployment but could not be set in a cephadm deployment out of the box due to the following reasons:
    1. The code changes enabled the user to define the CPU core for each OSD on each host but would also need the user to know the OSD ID being deployed on each host before deployment
    2. User cannot anticipate the location of OSD in the Crush map.
       Setting the cpu cores before OSD deployment is difficult due to the *unpredictability* of the allocation.

When upstream crimson image was picked up for testing few months ago, deployment has failed due the above mentioned reasons, to support cephadm deployments without reverting the new code, it was decided to come up with new config options which would enable user to do a "basic" deployment which may not yield the true power of Crimson OSD but would be enough to get started in the first place until we figure out a way to support the per OSD CPU core pinning through containerized deployment

Upstream tracker: https://tracker.ceph.com/issues/65752
Upstream PR(merged in main): https://github.com/ceph/ceph/pull/57593

Around June 2024, new parameters were introduced to support the said 'basic' deployment for Crimson OSD with PR - https://github.com/ceph/ceph/pull/57593
And the two newly introduced config options are:
    - crimson_seastar_num_threads
    - crimson_alien_op_num_threads

This PR(57593) is yet to be backported to Squid, and the first Crimson downstream build that was received on 09-Aug-2024 was based off of Squid branch and not 'main', consquently the required parameters are missing in this build.

Request is to backport the upstream PR(https://github.com/ceph/ceph/pull/57593) to Squid and trigger a new downstream Crimson build off of the latest Squid branch

>>> Until then containerized Crimson OSD deployment through Cephadm will remained blocked and so will any further testing

Version-Release number of selected component (if applicable):
ceph version 19.1.0-22.0.crimson.el9cp

How reproducible:
1/1

Steps to Reproduce:
1. Deploy ceph cluster with crimson image
    cephadm --image cp.stg.icr.io/cp/ibm-ceph/ceph-8-crimson-rhel9:latest bootstrap --mon-ip 172.20.20.21 --registry-json ~/crimson/ibm-registry.json --allow-fqdn-hostname --allow-overwrite --orphan-initial-daemons --initial-dashboard-password crimson-80-tp
2. Add hosts with labels
3. Deploy Monitors and Managers
4. Configure required parameters to enable crimson
  - # ceph config set global 'enable_experimental_unrecoverable_data_corrupting_features' crimson
  - # ceph osd set-allow-crimson --yes-i-really-mean-it
  - # ceph config set mon osd_pool_default_crimson true
5. Set the CPU core affinity at global OSD level
  - # ceph config set osd crimson_seastar_num_threads <N>
6. Deploy OSDs, either on all available devices or using spec file

Actual results:
Crimson OSD deployment is not possible in containerized deployments through cephadm until PR https://github.com/ceph/ceph/pull/57593 is backported to Squid

Expected results:
Crimson OSD deployment with necessary config parameters set should be possible with cephadm orchestrator

Additional info:

>>> Build version -
    [root@bruuni011 ubuntu]# cephadm shell -- ceph versions
    Inferring fsid fe1e8690-603e-11ef-a532-0cc47af96462
    Inferring config /var/lib/ceph/fe1e8690-603e-11ef-a532-0cc47af96462/mon.bruuni011/config
    Using ceph image with id '7d2a9d161de6' and tag 'latest' created on 2024-08-08 19:51:29 +0000 UTC
    cp.stg.icr.io/cp/ibm-ceph/ceph-8-crimson-rhel9@sha256:47d13be71934eb3a6d286333f8818628dce0f1bf43164a7d0b263d9840e5d300
    2024-08-23T12:57:54.715+0000 7f70e563e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson
    2024-08-23T12:57:54.716+0000 7f70e563e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson
    {
        "mon": {
            "ceph version 19.1.0-22.0.crimson.el9cp (e5b7dfedb7d8a66d166eb0f98361f71bdb7905ad) squid (rc)": 5
        },
        "mgr": {
            "ceph version 19.1.0-22.0.crimson.el9cp (e5b7dfedb7d8a66d166eb0f98361f71bdb7905ad) squid (rc)": 5
        },
        "overall": {
            "ceph version 19.1.0-22.0.crimson.el9cp (e5b7dfedb7d8a66d166eb0f98361f71bdb7905ad) squid (rc)": 10
        }
    }

>>> crimson_seastar_num_threads is unrecognized
  [root@bruuni011 ubuntu]# ceph config get osd crimson_seastar_num_threads
  2024-08-22T17:45:36.835+0000 7f64f0141640 -1 WARNING: the following dangerous and experimental features are enabled: crimson
  2024-08-22T17:45:36.835+0000 7f64f0141640 -1 WARNING: the following dangerous and experimental features are enabled: crimson
  Error ENOENT: unrecognized key 'crimson_seastar_num_threads'
  
  [root@bruuni011 ubuntu]# ceph config get osd crimson_alien_op_num_threads
  2024-08-22T17:45:54.723+0000 7f184748b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson
  2024-08-22T17:45:54.724+0000 7f184748b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson
  6

>>> Config dump
    [root@bruuni011 ubuntu]# ceph config dump
    2024-08-23T13:08:34.695+0000 7f7644f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson
    2024-08-23T13:08:34.696+0000 7f7644f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson
    WHO     MASK  LEVEL     OPTION                                                      VALUE                                                                                                                   RO
    global        basic     container_image                                             cp.stg.icr.io/cp/ibm-ceph/ceph-8-crimson-rhel9@sha256:47d13be71934eb3a6d286333f8818628dce0f1bf43164a7d0b263d9840e5d300  * 
    global        advanced  enable_experimental_unrecoverable_data_corrupting_features  crimson                                                                                                                   
    mon           advanced  auth_allow_insecure_global_id_reclaim                       false                                                                                                                     
    mon           advanced  osd_pool_default_crimson                                    true                                                                                                                      
    mon           advanced  public_network                                              172.20.20.0/24                                                                                                          * 
    mgr           advanced  mgr/cephadm/container_init                                  True                                                                                                                    * 
    mgr           advanced  mgr/cephadm/migration_current                               8                                                                                                                       * 
    mgr           advanced  mgr/dashboard/ALERTMANAGER_API_HOST                         http://bruuni011.back.ceph.redhat.com:9093                                                                              * 
    mgr           advanced  mgr/dashboard/GRAFANA_API_SSL_VERIFY                        false                                                                                                                   * 
    mgr           advanced  mgr/dashboard/GRAFANA_API_URL                               https://bruuni011.back.ceph.redhat.com:3000                                                                             * 
    mgr           advanced  mgr/dashboard/PROMETHEUS_API_HOST                           http://bruuni011.back.ceph.redhat.com:9095                                                                              * 
    mgr           advanced  mgr/dashboard/ssl_server_port                               8443                                                                                                                    * 
    mgr           advanced  mgr/orchestrator/orchestrator                               cephadm                                                                                                                   
    osd           advanced  osd_memory_target_autotune                                  true

>>> OSD log snippet

    [root@bruuni003 fe1e8690-603e-11ef-a532-0cc47af96462]# tail -25 ceph-osd.3.log 
    INFO  2024-08-23 12:37:31,125 [shard  0:main] ms - 0x3000827f98c0 client.?(temp_mon_client) 172.20.20.13:0/2760811900@50120 >> mon.? v2:172.20.20.18:3300/0 protocol CONNECTING execute_connecting is aborted at inconsistent CLOSING -- negotiation failure
    INFO  2024-08-23 12:37:31,125 [shard  0:main] ms - 0x3000827f9700 client.?(temp_mon_client) 172.20.20.13:0/2760811900@54376 >> mon.? v2:172.20.20.21:3300/0 protocol CONNECTING execute_connecting is aborted at inconsistent CLOSING -- negotiation failure
    WARN  2024-08-23 12:37:31,126 [shard  0:main] ms - 0x3000827f9540 client.?(temp_mon_client) 172.20.20.13:0/2760811900@53424 >> mon.2 v2:172.20.20.22:3300/0 UPDATE Policy(lossy=true) from server flags
    INFO  2024-08-23 12:37:31,126 [shard  0:main] ms - 0x3000827f9540 client.?(temp_mon_client) 172.20.20.13:0/2760811900@53424 >> mon.2 v2:172.20.20.22:3300/0 connected: gs=3, pgs=10349, cs=0, client_cookie=11050325961552303138, server_cookie=0, io(in_seq=0, is_out_queued=false, has_out_sent=false), new_sid=0, send 3 IOHandler::dispatch_connect()
    INFO  2024-08-23 12:37:31,126 [shard  0:main] ms - 0x3000827f9540 client.?(temp_mon_client) 172.20.20.13:0/2760811900@53424 >> mon.2 v2:172.20.20.22:3300/0 do_out_dispatch: stop(delay...) at delay, no out_exit_dispatching
    INFO  2024-08-23 12:37:31,126 [shard  0:main] ms - 0x3000827f9540 client.?(temp_mon_client) 172.20.20.13:0/2760811900@53424 >> mon.2 v2:172.20.20.22:3300/0 do_out_dispatch: stop(switched) at switched, no out_exit_dispatching
    INFO  2024-08-23 12:37:31,126 [shard  0:main] monc - got monmap 5, mon.bruuni012, is now rank 2
    INFO  2024-08-23 12:37:31,126 [shard  0:main] monc - handle_monmap: renewing tickets
    INFO  2024-08-23 12:37:31,126 [shard  0:main] monc - renew_rotating_keyring renewing rotating keys  (they expired before 1724416621.1261876)
    INFO  2024-08-23 12:37:31,126 [shard  0:main] monc - renew_rotating_keyring called too often (last: 1724416651.1257656)
    INFO  2024-08-23 12:37:31,126 [shard  0:main] monc - handle_mon_map: renewed tickets
    INFO  2024-08-23 12:37:31,126 [shard  0:main] monc - handle_auth_reply [0x3000827f9540 client.?(temp_mon_client) 172.20.20.13:0/2760811900@53424 >> mon.2 v2:172.20.20.22:3300/0] returns auth_reply(proto 2 0 (0) Success) v1: 0
    INFO  2024-08-23 12:37:31,126 [shard  0:main] monc - handle_auth_reply
    INFO  2024-08-23 12:37:31,126 [shard  0:main] monc - renew_rotating_keyring renewing rotating keys  (they expired before 1724416621.1262548)
    INFO  2024-08-23 12:37:31,126 [shard  0:main] monc - renew_rotating_keyring called too often (last: 1724416651.1257656)
    INFO  2024-08-23 12:37:31,126 [shard  0:main] monc - do_auth_single: [0x3000827f9540 client.?(temp_mon_client) 172.20.20.13:0/2760811900@53424 >> mon.2 v2:172.20.20.22:3300/0] returns auth_reply(proto 2 0 (0) Success) v1: 0
    WARN  2024-08-23 12:37:31,126 [shard  0:main] monc - renew_subs - empty
    INFO  2024-08-23 12:37:31,126 [shard  0:main] monc - set_mon_vals no callback set
    INFO  2024-08-23 12:37:31,127 [shard  0:main] monc - stop
    INFO  2024-08-23 12:37:31,127 [shard  0:main] monc - close
    INFO  2024-08-23 12:37:31,127 [shard  0:main] ms - 0x3000827f9540 client.?(temp_mon_client) 172.20.20.13:0/2760811900@53424 >> mon.2 v2:172.20.20.22:3300/0 mark_down() at io_stat(io_state=open, in_seq=3, out_seq=3, out_pending_msgs_size=0, out_sent_msgs_size=0, need_ack=0, need_keepalive=0, need_keepalive_ack=0), send 1 notify_mark_down()
    INFO  2024-08-23 12:37:31,127 [shard  0:main] ms - 0x3000827f9540 client.?(temp_mon_client) 172.20.20.13:0/2760811900@53424 >> mon.2 v2:172.20.20.22:3300/0 closing: reset no, replace no
    INFO  2024-08-23 12:37:31,127 [shard  0:main] ms - 0x3000827f9540 client.?(temp_mon_client) 172.20.20.13:0/2760811900@53424 >> mon.2 v2:172.20.20.22:3300/0 do_in_dispatch(): fault at drop, io_stat(io_state=drop, in_seq=3, out_seq=3, out_pending_msgs_size=0, out_sent_msgs_size=0, need_ack=0, need_keepalive=0, need_keepalive_ack=0) -- read eof
    ERROR 2024-08-23 12:37:31,129 [shard  0:main] none - /builddir/build/BUILD/ceph-19.1.0/src/crimson/os/alienstore/alien_store.cc:112 : In function 'virtual seastar::future<> crimson::os::AlienStore::start()', ceph_assert(%s)
    cpu_cores.has_value()