Terminology

Bare-Metal is a a physical computer and it’s base OS

Host is a bare-metal machine that runs VMs

Thread is a core of a CPU

CPU thread refers to the thread on the host system CPU

vCPU thread is a thread on a VM’s virtualized CPU

Note: I borrowed terminology heavily from this article on null-src

Isolating CPUs

My last article covers this topic. Isolating a VM’s vCPU threads from the host system ensures that host’s processes don’t interfere with the VM.

Scheduling Openstack VMs on isolated cores

CPU Affinity

using taskset

Linux Process Scheduler Tuning

GRUB cmdline

Okay… don’t delete the console entries in your command line… the entries below are for isolated cpu 1. This is not a comprehensive list… Just append whatever makes sense to you.

GRUB_CMDLINE_LINUX_DEFAULT="threadirqs rcu_nocbs=1 rcu_nocb_poll isolcpus=1 nohz_full=1 nohz=off intel_pstate=disable nosoftlockup nohalt"

• KubeEdge
• k3s and k3os
• StarlingX

I present my brief impressions of each below. This post will need updates as my research goes on.

Background

Edge is often used as a vague term for either one of CDN, factory network or IT data center. There is a fair amount of confusion where cloud transforms into “Edge”.

Fog apparently is used to describe the same concepts and maintains the vaporous water analogue that is so popular in this space (think about it: data lake, fog, clouds …) Usage of the term “Fog” has been tainted by Cisco. They make a poor attempt to show the difference. Meh.

kubernetes is a container orchestrator that has become extremely popular in the last few years. It provides a common API to manage all containers running in your system.

Cloud native is used to describe container-first and container-based systems. (my own definition)

CNCF is the cloud native computing foundation. It manages projects that specifically target cloud native systems. It does not manage all cloud-native projects. To the contrary, there is a small list of projects that are managed by CNCF.

Linux Foundation maintains a set of open source projects. LF Edge is the subgroup that does this for projects targeting Edge deployments. LF projects are of all types.

OpenStack Foundation is another foundation that was originally formed for OpenStack’s projects (next item) but it seems to now be accepting projects related to other systems.

OpenStack is the most popular open source cloud. It’s the OG thing: was popular before docker, k8s or anything container-related was mainstream. There are still many users of OpenStack in the wild and it’s not going anywhere soon.

KubeEdge

KubeEdge is a CNCF sandbox project designed to extend Kubernetes from cloud to edge. The IoT Edge has unique challenges that no one solution can provide a complete solution for. KubeEdge is designed to meet the challenge of:

• Network Reliability
• Resource constraint; and
• Scalability

KubeEdge has agents running on the Edge. The control plane resides on the Cloud and if there’s a network failure/disconnect, the agents keep on doing the necessary control operations. Metadata is synchronized as soon as the connection is restored.

KubeEdge is an opinionated take on the Edge: it provides services and frameworks for messaging and networking. And, since it’s opinionated, everyone might not agree with it.

My opinion is that KubeEdge will keep on serving a large userbase but it might not be extremely well suited to the 20% market that wants extra flexibility. How it pans out long-term is yet to be seen.

KubeEdge is bringing edge-edge communication which will be interesting combined with newer enhancements in the Linux kernel (read: Wireguard.) This is an extremely interesting project but for the Industry I’m working in, might not be the first choice - atleast for me.

k3s

K3s is a take on Kubernetes that removes lot’s of stuff off of standard kubernetes.

k3os

benefits of not maintaining your OS and it’s downsides

StarlingX

StarlingX is an integration project centered around Kubernetes. StarlingX makes an opinionated set of decisions for people looking to use Kubernetes for their infrastructure. StarlingX (STX) is part of the OpenStack foundation which is a testament to it’s high quality.

Using annotations, you can:

• Customize the Kernel that is used (see KernelPath example below)
• Change the guest image (ImagePath)
• Hypervisor (HypervisorPath)
• Firmware (FirmwarePath)
• etc. etc. (All of the annotations are listed in annotations.go)

Using a Custom Kernel

Here is the yaml file I used to apply a custom kernel:

apiVersion: v1
kind: Pod
metadata:
    annotations:
        com.github.containers.virtcontainers.KernelPath: "/usr/share/kata-containers/vmlinuz-4.19.31-40"
    name: kata-example
spec:
  runtimeClassName: kata-qemu

  containers:
  - name: nginx-container
    image: nginx

The metadata key contains the annotations map. All of the annotations are listed in annotations.go on the main repo.

This is documented here: https://github.com/kata-containers/packaging/blob/master/kata-deploy/README.md#docker-quick-start

Using Minikube and Kata

This PR is adding the documentation to get started with minikube: https://github.com/kata-containers/documentation/pull/445#

Using kubeadm

• First, set up a Kubernetes cluster using kubeadm: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
• then, https://github.com/kata-containers/packaging/blob/master/kata-deploy/README.md#kubernetes-quick-start

Conclusion

Using the new kata-deploy method, it is very easy to set up a Kubernetes cluster that uses Kata Containers.