STX on sysarcher/@ich_codehttps://sysarcher.github.io/categories/stx/Recent content in STX on sysarcher/@ich_codeHugoen-usMon, 09 Nov 2020 21:22:37 +0200Core Isolation Feature in STX4.0https://sysarcher.github.io/posts/stx-for-rt/Mon, 09 Nov 2020 21:22:37 +0200https://sysarcher.github.io/posts/stx-for-rt/<p><em>StarlingX is continuing to evolve and update it&rsquo;s Real-time capabilities but documentation can be a bit sparse. This page is hopefully a placeholder for until we have a great documentation around the capabilities.</em></p> <h3 id="isolating-cpu-cores">Isolating CPU Cores</h3> <p>In a StarlingX cluster CPU isolation and Kernel commandline parameters are (generally) managed from the active controller through <code>system host-cpu-XXX</code> commands.</p> <p>For isolation, we have the <code>Application-isolated</code> assigned_function.</p> <p>Get current functions:</p> <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ system host-cpu-list controller-0 </span></span><span style="display:flex;"><span>+--------------------------------------+----------+-----------+----------+--------+-----------------------------------------+-------------------+ </span></span><span style="display:flex;"><span>| uuid | log_core | processor | phy_core | thread | processor_model | assigned_function | </span></span><span style="display:flex;"><span>+--------------------------------------+----------+-----------+----------+--------+-----------------------------------------+-------------------+ </span></span><span style="display:flex;"><span>| 65b8337e-08bf-4e32-b7e6-17505eaf0806 | 0 | 0 | 0 | 0 | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Platform | </span></span><span style="display:flex;"><span>| b7578ba2-cba0-46cd-b860-3ba5c03ba91d | 1 | 0 | 1 | 0 | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Platform | </span></span><span style="display:flex;"><span>| ff7447a2-00df-43db-8716-d1416faf89bc | 2 | 0 | 2 | 0 | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application | </span></span><span style="display:flex;"><span>| 142796e5-b17d-48e9-b18a-b84c1ea5ae37 | 3 | 0 | 3 | 0 | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application | </span></span><span style="display:flex;"><span>| c68a8dc7-367f-4512-84ca-ec559fbb63f4 | 4 | 0 | 4 | 0 | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application | </span></span><span style="display:flex;"><span>| 15f360e2-d27c-4d9a-b0ff-0b78387eeb8c | 5 | 0 | 5 | 0 | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application | </span></span><span style="display:flex;"><span>| 89274cb5-4fb5-4f5b-a8c5-ec36394b5ce5 | 6 | 0 | 6 | 0 | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application | </span></span><span style="display:flex;"><span>| 33fb1e8e-77cd-4334-b061-dd40e6b7f7ba | 7 | 0 | 7 | 0 | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application | </span></span><span style="display:flex;"><span>| 8c804a31-c61e-408c-83c4-3c7192a861d8 | 8 | 0 | 0 | 1 | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Platform | </span></span><span style="display:flex;"><span>| 560ad09d-48df-42ec-8588-7e739030d7e3 | 9 | 0 | 1 | 1 | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Platform | </span></span><span style="display:flex;"><span>| cbd83713-0118-4995-9810-486cfdbb912e | 10 | 0 | 2 | 1 | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application | </span></span><span style="display:flex;"><span>| cb45713a-fa51-4628-b536-e3d5d57c891e | 11 | 0 | 3 | 1 | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application | </span></span><span style="display:flex;"><span>| 28dff3de-9913-4055-89d5-09dfc084c3c1 | 12 | 0 | 4 | 1 | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application | </span></span><span style="display:flex;"><span>| 98cf6c9e-2717-4f88-b487-487dbdadd18e | 13 | 0 | 5 | 1 | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application | </span></span><span style="display:flex;"><span>| 73dcec10-7a62-4d07-ba23-3482d0e0f2af | 14 | 0 | 6 | 1 | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application | </span></span><span style="display:flex;"><span>| 4a5a4477-f0db-4d77-9a6a-c77e5ca292e4 | 15 | 0 | 7 | 1 | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application | </span></span><span style="display:flex;"><span>+--------------------------------------+----------+-----------+----------+--------+-----------------------------------------+-------------------+ </span></span></code></pre></div><p>The <code>system host-cpu-modify</code> command is used to modify the assigned function.</p>StarlingX is continuing to evolve and update it’s Real-time capabilities but documentation can be a bit sparse. This page is hopefully a placeholder for until we have a great documentation around the capabilities.

Isolating CPU Cores

In a StarlingX cluster CPU isolation and Kernel commandline parameters are (generally) managed from the active controller through system host-cpu-XXX commands.

For isolation, we have the Application-isolated assigned_function.

Get current functions:

$ system host-cpu-list controller-0
+--------------------------------------+----------+-----------+----------+--------+-----------------------------------------+-------------------+
| uuid                                 | log_core | processor | phy_core | thread | processor_model                         | assigned_function |
+--------------------------------------+----------+-----------+----------+--------+-----------------------------------------+-------------------+
| 65b8337e-08bf-4e32-b7e6-17505eaf0806 | 0        | 0         | 0        | 0      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Platform          |
| b7578ba2-cba0-46cd-b860-3ba5c03ba91d | 1        | 0         | 1        | 0      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Platform          |
| ff7447a2-00df-43db-8716-d1416faf89bc | 2        | 0         | 2        | 0      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 142796e5-b17d-48e9-b18a-b84c1ea5ae37 | 3        | 0         | 3        | 0      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| c68a8dc7-367f-4512-84ca-ec559fbb63f4 | 4        | 0         | 4        | 0      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 15f360e2-d27c-4d9a-b0ff-0b78387eeb8c | 5        | 0         | 5        | 0      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 89274cb5-4fb5-4f5b-a8c5-ec36394b5ce5 | 6        | 0         | 6        | 0      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 33fb1e8e-77cd-4334-b061-dd40e6b7f7ba | 7        | 0         | 7        | 0      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 8c804a31-c61e-408c-83c4-3c7192a861d8 | 8        | 0         | 0        | 1      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Platform          |
| 560ad09d-48df-42ec-8588-7e739030d7e3 | 9        | 0         | 1        | 1      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Platform          |
| cbd83713-0118-4995-9810-486cfdbb912e | 10       | 0         | 2        | 1      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| cb45713a-fa51-4628-b536-e3d5d57c891e | 11       | 0         | 3        | 1      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 28dff3de-9913-4055-89d5-09dfc084c3c1 | 12       | 0         | 4        | 1      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 98cf6c9e-2717-4f88-b487-487dbdadd18e | 13       | 0         | 5        | 1      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 73dcec10-7a62-4d07-ba23-3482d0e0f2af | 14       | 0         | 6        | 1      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 4a5a4477-f0db-4d77-9a6a-c77e5ca292e4 | 15       | 0         | 7        | 1      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
+--------------------------------------+----------+-----------+----------+--------+-----------------------------------------+-------------------+

The system host-cpu-modify command is used to modify the assigned function.

$ system host-cpu-modify
usage: system host-cpu-modify [--nowrap] -f <function>
                              [-p0 <num_cores_on_processor0>]
                              [-p1 <num_cores_on_processor1>]
                              [-p2 <num_cores_on_processor2>]
                              [-p3 <num_cores_on_processor3>]
                              <hostnameorid>

For example, to set the 15th logical core to Application-isolated:

$ system host-cpu-modify -f Application-isolated -p0 15 controller-1

stx-openstack

One application that can make use of core isolation is stx-openstack. For example, if there are realtime VMs that require scheduling on isolated/dedicated cores, we need to set the compute key in the helm overrides (see OpenStack docs for: cpu_dedicated_set and cpu_shared_set).

Create a file for nova overrides that looks like this:

conf:
  overrides:
    nova_compute:
      hosts:
      - conf:
          # DELETED FOR CONCISENESS
        name: controller-0
      - conf:
          # DELETED FOR CONCISENESS
        name: controller-1
      - conf:
          nova:
            DEFAULT:
              # NO CHANGE
            compute:
              cpu_dedicated_set: '"1-2"'
              cpu_shared_set: '"3-5"'
            libvirt:
              # NO CHANGE
            pci:
              passthrough_whitelist:
                # NO CHANGE
            vnc:
              # NO CHANGE
        name: worker

Then, run the following command to apply these overrides:

$ # Update Helm overrides
$ system helm-override-update --values <yaml-file-with-overrides>.yaml stx-openstack nova openstack
$
$ # Apply application!!
$ system application-apply stx-openstack

Now, Openstack is able to schedule VMs on isolated CPU cores.

OpenStack Flavor

The following properties should be added to the RT VM (hugepages not yet included)

$ openstack flavor show rt.medium
+----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                      | Value                                                                                                                                                           |
+----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                                                                                                                                           |
| OS-FLV-EXT-DATA:ephemeral  | 0                                                                                                                                                               |
| access_project_ids         | None                                                                                                                                                            |
| disk                       | 10                                                                                                                                                              |
| id                         | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx                                                                                                                            |
| name                       | rt.medium                                                                                                                                                       |
| os-flavor-access:is_public | True                                                                                                                                                            |
| properties                 | hw:cpu_policy='dedicated', hw:cpu_realtime='yes', hw:cpu_realtime_mask='^0', hw:cpu_thread_policy='isolate', hw:emulator_threads_policy='share', hw:pmu='False' |
| ram                        | 2048                                                                                                                                                            |
| rxtx_factor                | 1.0                                                                                                                                                             |
| swap                       |                                                                                                                                                                 |
| vcpus                      | 2                                                                                                                                                               |
+----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+

Note on 2 vcpus: one will be used by linux and the other will be utilized by the RT program (cyclictest for instance)… You may use any number of vcpus … just make sure that one vcpu is dedicated for linux.

]]>