Not for prod but here’s a resolution:

Install Metrics Server

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.6.4/components.yaml

Apply the patch:

kubectl -n kube-system patch deployment metrics-server --type=json \
-p='[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--kubelet-insecure-tls"}]]'

thanks to @awalker125

Isolating CPU Cores

In a StarlingX cluster CPU isolation and Kernel commandline parameters are (generally) managed from the active controller through system host-cpu-XXX commands.

For isolation, we have the Application-isolated assigned_function.

Get current functions:

$ system host-cpu-list controller-0
+--------------------------------------+----------+-----------+----------+--------+-----------------------------------------+-------------------+
| uuid                                 | log_core | processor | phy_core | thread | processor_model                         | assigned_function |
+--------------------------------------+----------+-----------+----------+--------+-----------------------------------------+-------------------+
| 65b8337e-08bf-4e32-b7e6-17505eaf0806 | 0        | 0         | 0        | 0      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Platform          |
| b7578ba2-cba0-46cd-b860-3ba5c03ba91d | 1        | 0         | 1        | 0      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Platform          |
| ff7447a2-00df-43db-8716-d1416faf89bc | 2        | 0         | 2        | 0      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 142796e5-b17d-48e9-b18a-b84c1ea5ae37 | 3        | 0         | 3        | 0      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| c68a8dc7-367f-4512-84ca-ec559fbb63f4 | 4        | 0         | 4        | 0      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 15f360e2-d27c-4d9a-b0ff-0b78387eeb8c | 5        | 0         | 5        | 0      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 89274cb5-4fb5-4f5b-a8c5-ec36394b5ce5 | 6        | 0         | 6        | 0      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 33fb1e8e-77cd-4334-b061-dd40e6b7f7ba | 7        | 0         | 7        | 0      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 8c804a31-c61e-408c-83c4-3c7192a861d8 | 8        | 0         | 0        | 1      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Platform          |
| 560ad09d-48df-42ec-8588-7e739030d7e3 | 9        | 0         | 1        | 1      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Platform          |
| cbd83713-0118-4995-9810-486cfdbb912e | 10       | 0         | 2        | 1      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| cb45713a-fa51-4628-b536-e3d5d57c891e | 11       | 0         | 3        | 1      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 28dff3de-9913-4055-89d5-09dfc084c3c1 | 12       | 0         | 4        | 1      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 98cf6c9e-2717-4f88-b487-487dbdadd18e | 13       | 0         | 5        | 1      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 73dcec10-7a62-4d07-ba23-3482d0e0f2af | 14       | 0         | 6        | 1      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 4a5a4477-f0db-4d77-9a6a-c77e5ca292e4 | 15       | 0         | 7        | 1      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
+--------------------------------------+----------+-----------+----------+--------+-----------------------------------------+-------------------+

The system host-cpu-modify command is used to modify the assigned function.

$ system host-cpu-modify
usage: system host-cpu-modify [--nowrap] -f <function>
                              [-p0 <num_cores_on_processor0>]
                              [-p1 <num_cores_on_processor1>]
                              [-p2 <num_cores_on_processor2>]
                              [-p3 <num_cores_on_processor3>]
                              <hostnameorid>

For example, to set the 15th logical core to Application-isolated:

$ system host-cpu-modify -f Application-isolated -p0 15 controller-1

stx-openstack

One application that can make use of core isolation is stx-openstack. For example, if there are realtime VMs that require scheduling on isolated/dedicated cores, we need to set the compute key in the helm overrides (see OpenStack docs for: cpu_dedicated_set and cpu_shared_set).

Create a file for nova overrides that looks like this:

conf:
  overrides:
    nova_compute:
      hosts:
      - conf:
          # DELETED FOR CONCISENESS
        name: controller-0
      - conf:
          # DELETED FOR CONCISENESS
        name: controller-1
      - conf:
          nova:
            DEFAULT:
              # NO CHANGE
            compute:
              cpu_dedicated_set: '"1-2"'
              cpu_shared_set: '"3-5"'
            libvirt:
              # NO CHANGE
            pci:
              passthrough_whitelist:
                # NO CHANGE
            vnc:
              # NO CHANGE
        name: worker

Then, run the following command to apply these overrides:

$ # Update Helm overrides
$ system helm-override-update --values <yaml-file-with-overrides>.yaml stx-openstack nova openstack
$
$ # Apply application!!
$ system application-apply stx-openstack

Now, Openstack is able to schedule VMs on isolated CPU cores.

OpenStack Flavor

The following properties should be added to the RT VM (hugepages not yet included)

$ openstack flavor show rt.medium
+----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                      | Value                                                                                                                                                           |
+----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                                                                                                                                           |
| OS-FLV-EXT-DATA:ephemeral  | 0                                                                                                                                                               |
| access_project_ids         | None                                                                                                                                                            |
| disk                       | 10                                                                                                                                                              |
| id                         | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx                                                                                                                            |
| name                       | rt.medium                                                                                                                                                       |
| os-flavor-access:is_public | True                                                                                                                                                            |
| properties                 | hw:cpu_policy='dedicated', hw:cpu_realtime='yes', hw:cpu_realtime_mask='^0', hw:cpu_thread_policy='isolate', hw:emulator_threads_policy='share', hw:pmu='False' |
| ram                        | 2048                                                                                                                                                            |
| rxtx_factor                | 1.0                                                                                                                                                             |
| swap                       |                                                                                                                                                                 |
| vcpus                      | 2                                                                                                                                                               |
+----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+

Note on 2 vcpus: one will be used by linux and the other will be utilized by the RT program (cyclictest for instance)… You may use any number of vcpus … just make sure that one vcpu is dedicated for linux.

• ESC: I have remapped the Escape key to CAPSLOCK. Who uses CapsLock? ESC on the other hand is a Vim user’s go-to reflex. So keep it close-by.
• ESC+HJKL: For navigation. Why would you use muscles to move over to the arrow keys when you have them under your fingers.

How-to

xmodmap can be used to modify these key settings. But I think it will only work for X-sessions.

I found dconf/org/gnome/desktop/input-sources/xkb-options to be a much more solid solution.

Under xkb-options you can put a “Custom value” like: ['caps:swapescape'] (note: syntax is important!!)

Windows Note

By far the best and most reliable key remapper on Windows is sharpkeys (randyrants/sharpkeys).

If you’re not familiar with WebAssembly, I would suggest you start here:

• A cartoon intro to WebAssembly
• WebAssembly Concepts

This blog post is about wondering what WebAssembly can look like for some usecases.

Run Everywhere

WebAssembly can run everywhere - atleast that’s the goal. It’s performant - as performant as C/C++ and it’s secure.

WebAssembly is a type of code that can be run in modern web browsers and provides new features and major gains in performance.

https://developer.mozilla.org/en-US/docs/WebAssembly/Concepts

Open Questions

Although designed for the web, WebAssembly is able to run portably across architectures. This can be a nice way to distribute applications. Some day perhaps WebAssembly runtimes will be a part of all OSes.

What needs to be determined IMO is:

• Realtime capabilities

  Can WebAssembly modules written in C/C++ provide any RT capabilities? If so, then it’ll be interesting to evaluate whether a WebAssembly runtime is all that a PLC should be. Apps running the PLC would be delivered like software. Less stickiness for everybody.

• Recompile all the things

  Can WebAssembly runtimes replace native programs? How about message queues, brokers implemented in WebAssembly? I know it’s not possible to compile all complicated software to WebAssembly today - but is that the future too?

• UI

  Will Slack and it’s likes re-implement significant portions of their code to WebAssembly? Should work, right?

• Edge

  Fastly has already done lots of work with their CDN running WebAssembly-isolated workloads. Can we bring this down to the Edge where we’re running containers today?

Conclusion

There’re so many things I’m looking an answer for. This year is going to be awesome!

$ sudo apt install python3 python3-pip    # installs pip3
$ sudo pip3 install --upgrade pip         # upgrade pip to latest version

Congrats, now you’re stuck with whatever version ships with the distro. I think we can do better!

pyenv

My recommendation now, is to use pyenv with which you can manage multiple versions, virtualenvs through a nice CLI

Uninstall pip/pip3

Before proceeding, I’d suggest uninstalling pip using pip uninstall pip and sudo apt remove python3-pip

Installation

Using the pyenv installer project

$ sudo apt install libssl-dev \
                   libsqlite-dev \
                   libreadline-dev \
                   libbz2-dev  \
                   libsqlite3-dev  # I've observed failures without these libs
$ curl https://pyenv.run | bash
...
WARNING: seems you still have not added 'pyenv' to the load path.

# Load pyenv automatically by adding
# the following to ~/.bashrc:

export PATH="/home/taimoor/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

This should set up pyenv for you. Read the instructions printed at the end and follow them! Add the three things to your ~/.bashrc or ~/.zshrc or whatever.

Usage (summarized)

Example: Install Python 3.8.1:

$ pyenv install 3.8.1

P.S: logout and log back in

P.S2: Tab completion works!

Virtualenv

Create a virtualenv

$ pyenv virtualenv 3.8.1 venv3.8    # create a virtualenv called venv3.8 based on python 3.8.1
$ pyenv activate venv3.8    # activate venv3.8
$ python --version
Python 3.8.1
$ which python
/home/taimoor/.pyenv/shims/python

That’s it! Now you can manage your anaconda and python environments, versions sooo much better.

2020 used to be hindsight but not any more. It’s here. I’ll write down my own resolutions for this year here…

Technology

• Continue with my FP journey (Clojure and F#)
• Keep on learning Rust and Golang
• (Maybe) look into Ziglang
• DDD and Software Architecture. I have been reading a lot about it recently and will continue to try to do so. Especially Wlachsinn’s book!

Health

• Start exercising regularly
• Eat healthy food normally (it’s fine to eat unhealthy sometimes)
• Swim!

Misc

• Start a side-project in Rust (embedded??)
• Start a web Frontend (Clojurescript??) project

Conclusion

I’m sure I’ll miss out on some things and do some right… Fingers crossed.

I was part of a team that presented the concept of OT/IT consolidation. I got quite a few comments and questions about the demo video (in Urdu here). So here’s a shot at answering some of the questions I got forwarded:

The demo was (in oversimplified terms) showing PLC compute moved to software which can be orchestrated using Kubernetes.

Q/A

• Nice move. But how sensor data is being processed without PLC or a microcontroller?

There are “dumb” IO devices connected over Profinet in this case (but it can really be any fieldbus e.g. Ethercat etc.)

Some people had a comment about ADCs being involved. YES, but we need to think at a higher level in this case. Analog and Digital IOs are obviously involved under-the-hood.

• So what is difference between an arduino microcontroller and program able logical controller

This is a strange question. Arduino is not a microcontroller, it’s a prototyping platform (and a company as well.) This prototyping platform has a microcontroller. A PLC is a digital computer made for the industry. Follows many industrial IEC standards. It is a part of many things. For our demo, we were using ADAM PLCs from Advantech.

Arduino can also be programmed to become a PLC (see the OpenPLC project: https://www.openplcproject.com/)

• Ok good step so what precautions u take if ur factory text file or source code get hack??

If you get hacked, “precautions” are irrelevant. The word precaution means that it ought to prevent the hack. I’m assuming this was the question as well. Although this was not a security demo, but k8s brings with it good security practices (certs, https etc.). “Hack” is too open-ended of a word.

Security practices are a whole different topic. Perhaps not relevant for the discussion here.

• So can we apply machine learning and Ai codes in PLC??

I would read my answer above and conclude: NO. PLCs are meant for other things, Google is your friend!

• We get machine learning codes in Python mostly some standard c language so what u suggest??

Great attempt at moving off-topic! (Always one person in the team!!)

• Using structure text and block diagrams and ladder diagrams??

Yup

• One more thing can u tell me PLC have a capability of processing i mean controller and processor r 2 different things so can u guide me ?

They are computers in the end! The scope of their work is a bit limited to their application. Check out the OpenPLC project I linked to above.

• No. That is why I am designing machines and control systems using hybrid model. Means using PLCs, Microcontrollers and Computers to accomplish all kind of customization which include AI and custom GUIs in python and VueJs

Interesting… But we should not convolute so many things in one sentence. Everything you mentioned has its place. And they work together. We ought to avoid the use of the words “Hybrid Model” which sounds like formalism.

• Recently I have designed Pakistan’s first industrial Artificially intelligent sorting robotic conveyor using this hybrid model.

Tall claims!! I saw the video and one comment for now is that it might not meet some safety standards we have in the EU and US. Best of luck my friend!! Bring us laurels.

Also, I would tone down on Artificial intelligence and use specifics when talking to more technical folk (e.g. Computer Vision, OpenVINO, Inference, Learning etc.)

• So its coslty???

Yes, it’s supposed to run for many years so, there’s a lot of hardening.

Conclusion

I’m surprised by the interest people showed in the project. It’s still a concept and we want to work with the Industry leaders to bring about more mature products.

Getting started

Clojurescript is written in Clojure which requires Java. So we need to get all three up and running.

Java

To get Java JDK and JRE, run:

$ sudo apt install openjdk-11-jre
$ sudo apt install openjdk-11-jdk

For proxy settings, edit /etc/java-11-openjdk/net.properties.

Example for http proxy in the file:

http.proxyHost="proxy-addr.company.com"
http.proxyPort=8080
http.nonProxyHosts=localhost|127.*|[::1]

Maven

This was the hardest to figure out. And it’s crazy (coming from a newbie perspective) that maven doesn’t make it easy to just reuse Java or System proxies or respect the http_proxy variable. Madness. Moreover, you need to create the configuration file at ~/.m2/settings.xml with the following contents:

<settings>
<proxies>
    <proxy>
        <id>httpproxy</id>
        <active>true</active>
        <protocol>http</protocol>
        <host>proxy-addr.company.com</host>
        <port>8080</port>
        <nonProxyHosts>local.net|some.host.com</nonProxyHosts>
    </proxy>
    <proxy>
        <id>httpsproxy</id>
        <active>true</active>
        <protocol>https</protocol>
        <host>proxy-addr.company.com</host>
        <port>8080</port>
        <nonProxyHosts>local.net|some.host.com</nonProxyHosts>
    </proxy>
</proxies>
</settings>

Clojure

make sure rlwrap is installed (sudo apt install rlwrap)

Download and install clojure using the following commands:

$ curl -O https://download.clojure.org/install/linux-install-1.10.1.469.sh  # This is the current version as of writing
$ chmod +x linux-install-1.10.1.469.sh
$ sudo ./linux-install-1.10.1.469.sh

Using the command clj launches the repl. Thankfully!

Leiningen

Follow the steps on https://leiningen.org/

curl -O "https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein"
chmod a+x lein
mv lein ~/bin/

Create a Test Project

To test lein and figwheel+reagent:

$ lein new figwheel some-name -- --reagent

• Big ball of mud
• Monoliths
• Event-driven architectures
• Service-oriented architectures (SOA)
• Serverless

So I got to play with the Data Plane Development Kit (DPDK) from Intel recently. I planned on replicating the PVP benchmark for my use-case. I set up the following environment:

• Ubuntu 18.04 (bionic) host with:
  • KVM
  • QEMU
• Debian 10 guest images downloaded from: https://cdimage.debian.org/cdimage/openstack/current/

A useful tool is numactl (use apt to install it). On my non-server system, I don’t have multiple socks and multiple cores per sock etc.

root@machine:~# numactl  -s
policy: default
preferred node: current
physcpubind: 2 3 4 5 6 7 
cpubind: 0 
nodebind: 0 
membind: 0 

Setting up (OVS+DPDK)

First of all set install OVS-dpdk using apt

sudo apt-get install openvswitch-switch-dpdk 
sudo update-alternatives --set ovs-vswitchd /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk

Restart the systemd service: sudo systemctl restart openvswitch-switch.service

Hugepages will need to be set up. But we need to tell DPDK about that.

root@machine:~# cat /etc/dpdk/dpdk.conf
   ...
NR_1G_PAGES=4
   ...

Isolate some CPUs

Edit the value of GRUB_CMDLINE_LINUX_DEFAULT to:

GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=4 hugepagesz=2M hugepages=2048 iommu=pt intel_iommu=on isolcpus=0,1"

And run sudo update-grub.

this setting will not need to change afterwards. A reboot of the machine is required. See if the parameters got passed correctly by doing cat /proc/cmdline

Set up Hugepages

sudo mkdir -p /mnt/huge
sudo mkdir -p /mnt/huge_2mb
sudo mount -t hugetlbfs none /mnt/huge
sudo mount -t hugetlbfs none /mnt/huge_2mb -o pagesize=2MB
sudo mount -t hugetlbfs none /dev/hugepages

root@machine:~# grep HugePages_ /proc/meminfo 
HugePages_Total:       4
HugePages_Free:        1
HugePages_Rsvd:        0
HugePages_Surp:        0

Configure OVS-DPDK

The database needs to be set up only once (sudo ovs-vsctl --no-wait init)

sudo ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0x03  # hopefully: cores, 0,1 
sudo ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="1024"
sudo ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x0C # we have just one NUMA node

Create OVS-DPDK Bridge and Ports

dpdkvhostuser is a predefined type. vhost-user is understood by Qemu 2.2 and above.

sudo ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
sudo ovs-vsctl add-port br0 vhost-user1 -- set Interface vhost-user1 type=dpdkvhostuser
sudo ovs-vsctl add-port br0 vhost-user2 -- set Interface vhost-user2 type=dpdkvhostuser

root@machine:~# ovs-vsctl show
c3459670-39b8-43f7-bc57-569672ed79c6
    Bridge "br0"
        Port "br0"
            Interface "br0"
                type: internal
        Port "vhost-user2"
            Interface "vhost-user2"
                type: dpdkvhostuser
        Port "vhost-user1"
            Interface "vhost-user1"
                type: dpdkvhostuser
    ovs_version: "2.9.2"

Binding Devices to DPDK

We should check whether the NIC is compatible with DPDK first. The list on DPDK doesn’t seem to have been updated e.g. the documentation itself for newer versions states that the I210 is supported.

sudo modprobe vfio-pci
sudo dpdk-devbind --bind=vfio-pci eno1 # eno1 is the interface I want to bind to DPDK

dpdk-devbind --status can be used to check whether the NIC is now using the DPDK compatible driver.

root@machine:~# dpdk-devbind --status

Network devices using DPDK-compatible driver
============================================
0000:00:19.0 'Ethernet Connection I217-V 153b' drv=vfio-pci unused=e1000e

Network devices using kernel driver
===================================
0000:02:00.0 'Killer E220x Gigabit Ethernet Controller e091' if=enp2s0 drv=alx unused=vfio-pci *Active*

    ...

VMs Quick Start

Download Debian 10 (buster) and change the password for root using guestfish. (Debian was an arbitrary choice here.)

Add anything to the images at this point. I downloaded qperf with all it’s dependencies and added it:

PACKAGES="qperf iperf"
apt-get download $(apt-cache depends --recurse --no-recommends --no-suggests \
  --no-conflicts --no-breaks --no-replaces --no-enhances \
  --no-pre-depends ${PACKAGES} | grep "^\w")

virt-copy-in -a ./debian-10.0.0-openstack-amd64-2.qcow2  *.deb  /home/

openssl passwd -1 <password> can be used to generate a password entry, and this can be added to the guest image using:

guestfish --rw --add ../debian-10.0.0-openstack-amd64.qcow2 --mount /dev/sda1:/ vi /etc/shadow

And make 2 copies of the debian-10.0.1-20190708-openstack-amd64.qcow2 file. We’ll use it as our hard drives for guests.

Some resources to read

• File-sharing: https://www.linux-kvm.org/page/9p_virtio
• OVS+KVM: https://docs.paloaltonetworks.com/vm-series/8-1/vm-series-deployment/set-up-the-vm-series-firewall-on-kvm/performance-tuning-of-the-vm-series-for-kvm/enable-open-vswitch-on-kvm.html#
• Using testpmd to test DPDK Performance: https://software.intel.com/en-us/articles/testing-dpdk-performance-and-features-with-testpmd
• Vhost/Virtio in DPDK: https://software.intel.com/en-us/articles/configuration-and-performance-of-vhost-virtio-in-data-plane-development-kit-dpdk

Launching the Guests

I launched the guests in two separate terminal windows (i.e., copy-paste instead of executing).

root@machine:~# cat ./launch-os.sh 
# Launch OS1
qemu-system-x86_64 -m 1024 -smp 4 -cpu host -hda ./debian-10.0.0-openstack-amd64.qcow2 -boot c -enable-kvm -no-reboot -net none -nographic -chardev socket,id=char1,path=/run/openvswitch/vhost-user1 -netdev type=vhost-user,id=mynet1,chardev=char1,vhostforce -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1 -object memory-backend-file,id=mem,size=1G,mem-path=/dev/hugepages,share=on -numa node,memdev=mem -mem-prealloc

# Launch OS2
qemu-system-x86_64 -m 1024 -smp 4 -cpu host -hda ./debian-10.0.0-openstack-amd64-2.qcow2 -boot c -enable-kvm -no-reboot -net none -nographic -chardev socket,id=char2,path=/run/openvswitch/vhost-user2 -netdev type=vhost-user,id=mynet2,chardev=char2,vhostforce -device virtio-net-pci,mac=00:00:00:00:00:02,netdev=mynet2 -object memory-backend-file,id=mem,size=1G,mem-path=/dev/hugepages,share=on -numa node,memdev=mem -mem-prealloc -fsdev local,security_model=passthrough,id=fsdev0,path=/tmp/share -device virtio-9p-pci,id=fs0,fsdev=fsdev0,mount_tag=hostshare

Godspeed!

Using annotations, you can:

• Customize the Kernel that is used (see KernelPath example below)
• Change the guest image (ImagePath)
• Hypervisor (HypervisorPath)
• Firmware (FirmwarePath)
• etc. etc. (All of the annotations are listed in annotations.go)

Using a Custom Kernel

Here is the yaml file I used to apply a custom kernel:

apiVersion: v1
kind: Pod
metadata:
    annotations:
        com.github.containers.virtcontainers.KernelPath: "/usr/share/kata-containers/vmlinuz-4.19.31-40"
    name: kata-example
spec:
  runtimeClassName: kata-qemu

  containers:
  - name: nginx-container
    image: nginx

The metadata key contains the annotations map. All of the annotations are listed in annotations.go on the main repo.

This is documented here: https://github.com/kata-containers/packaging/blob/master/kata-deploy/README.md#docker-quick-start

Using Minikube and Kata

This PR is adding the documentation to get started with minikube: https://github.com/kata-containers/documentation/pull/445#

Using kubeadm

• First, set up a Kubernetes cluster using kubeadm: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
• then, https://github.com/kata-containers/packaging/blob/master/kata-deploy/README.md#kubernetes-quick-start

Conclusion

Using the new kata-deploy method, it is very easy to set up a Kubernetes cluster that uses Kata Containers.

For now, let’s look at how I went about doing what I did..

1. You need to go get github.com/kata-containers/packaging repository. (or, git clone if you so prefer.) Be sure to have your $GOPATH properly set!!
2. cd into packaging/kernel directory.
3. I chose the 4.19.31 version of the kernel. Primarily because it was a newer kernel (not too different from the kernel version that comes with the current kata release.) and because it had the RT patches available
4. I changed the kernel_version="" line to kernel_version="4.19.31"

That’s it. We are ready to (1) apply the necessary patches from Kata maintainers to our Kernel of choosing and (2) apply the RT patches:

# download and apply the kata patches to the kernel
./build-kernel.sh setup

cd kata-linux-4.19.31-36
patch -p1<../path-to-extracted-patch/patch-4.19.31-rt18.patch

Wait… we need to first, make menuconfig and enable RT capabilities

General Setup 
  --> Preemption Model
    --> Fully Preemptible Kernel (RT)

Now, we’re ready to build the Kernel and install it*:

./build-kernel.sh setup
sudo ./build-kernel.sh install

Drum Roll please

$ docker run -it --runtime kata-runtime ubuntu bash
root@77cf04fd8290:/# uname -r
4.19.31-rt18
root@77cf04fd8290:/# uname -a
Linux 77cf04fd8290 4.19.31-rt18 #2 SMP PREEMPT RT Mon Apr 29 09:06:08 CEST 2019 x86_64 x86_64 x86_64 GNU/Linux
root@77cf04fd8290:/# 

So proud!!!

Happy Building!!

Note on Kernel Files

vmlinux: uncompressed kernel. | vmlinuz: compressed kernel. | bzImage: located under arch/x86_64/boot/bzImage. Big Zimage. Compressed kernel image.

More info: https://stackoverflow.com/a/22338835/3760442

Acknowledgements

Special thanks to @egernst, @rico and the great team on Slack!! Though this was my first foray into Kata Containers, I must say the folks were very welcoming. Thank you all!!

* I built the kernel (as of the time of writing) using make build -j4 instead of using the build-kernel.sh script but it shouldn’t matter. Just FYI.

Conclusion

What I concluded from the documentations is that we don’t get something ready-to-go from the official docs. And you should consider yourself lucky if everything worked for you out-of-the-box.

Setup Ideas

• Start from a clean installation of Ubuntu 18.04.
• The local.conf goes into the top-level directory of the devstack checkout.

Setup

My setup already provided the DNS and DHCP servers as illustrated below.

        (DNS Server)              (DHCP Server)
             |                          |
             |                          |
             +------------+-------------+
                          |
                          |
                        (OpenStack)

So, my local.conf looks like the following:

[[local|localrc]]
ADMIN_PASSWORD=safeandsecure
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD

HOST_IP=<HOST_IP>

FLOATING_RANGE=<HOST_IP>/24
Q_FLOATING_ALLOCATION_POOL=start=<HOST_IP>.150,end=<HOST_IP>.200
FIXED_RANGE=10.11.12.0/24
FIXED_NETWORK_SIZE=256
PUBLIC_INTERFACE=eno1
PUBLIC_NETWORK_GATEWAY=<HOST_IP>.1

LOGFILE=$DEST/logs/stack.sh.log
LOGDAYS=2

SWIFT_HASH=66a3d6b56c1f479c8b4e70ab5c2000f5
SWIFT_REPLICAS=1
SWIFT_DATA_DIR=$DEST/data

(The <HOST_IP>.XXX numbers are obviously replacing the last part of the IP address.)

What I’m telling OpenStack is that I …

• already have a Gateway in the network, use it!
• Already know what the IP allocation pool looks like

Next

With that, hopefully you can go do OpenStack stuff and not have to worry about networking being the biggest issue in your life. Good luck!!!

Benefits

• The “environment” you’re programming against are the function arguments
• Reuse is imminent. Composition is what everyone calls it.
• Functions’ results can be cached (memoization)

There must be many more benefits.. These three are the ones I can think of and the ones that get the most marks from my side.

Main Concepts

Functional programming is a technique that depends on the programmer adhering to some principles (it helps if the programming language forces him as well):

Purity

A pure function has no side effects i.e. it depends only on its arguments.

(defn pure-function [input]
  (if input output-1 output-2))

In this example, if the pure-function is really pure, then depending on the input, the return value always remains the same. This also means that the output-1 and output-2 data are immutable i.e. they’re unchangable!

Referential Transparency

In the example above, we might just have replaced the output of pure-funtion with either output-1 or output-2 thus avoiding the call to a function. Many languages can optimize this behavior by caching the result. It’s called memoization. This also means that you could parallelize stuff i.e. functions can be independently evaluated on different machines/cores/nodes whatevers.

Persistence

Immutability is key. You never mutate. An operation on data doesn’t change the data, it just returns new data. This naive approach is memory hungry I suppose but platforms and languages find ways to optimize stuff.

Laziness

Since I’m learning Clojure, there might be a bias because seq are everywhere and they’re laaaaaaaaaaazy! That is, you only evaluate stuff when it’s required and never before. This enables infinite length sequences. Also, we pay for operations only when required.

Conclusion

I’m still a Functional Programming newbie but Clojure is opening my eyes! I am intrigued.

Fin!

Lightning Talk

I did a lightning talk around Event Driven Microservices. Event driven architectures are becoming fairly common these days and IoT is a natural and easily understandable use-case. I presented the architecture we developed but without going into too much detail.

One question I had some trouble answering was around the “infinite-length” event streams. What if the queues get filled up.. How do we save state? The answers to these questions is it depends most of the time but usually you don’t run out of space at all :). And I refer to Martin Fowler for these questions.

Demo

I did an “Fearless Systems Programming with Rust” tutorial/demo as well. It was all going well… I presented a PowerPoint slide deck with some introductory stuff. And then “let me switch to a REAL OPERATING SYSTEM” said I. And I realized that I had a 13" Dell XPS. D’oh!! It only had a Thunderbolt connection and it wouldn’t be able to connect to the projector.

Then, I braved up.. Went unprepared into my Lab machine somewhere else and starting setting up a new project that did my container-demo style version of containers in Rust. Basically, going through the basics of containers/docker and writing those in Rust. (Thanks again Liz!)

Because I was now doing live coding without a prepared machine over ssh across Germany (lag and all ..), the demo didn’t go too smooth! But in the end I was proud that we did get the final product right and in the process we did (literally) demonstrate the power of the Rust compiler, and the Cargo toolchain.

Conclusions

• Take all the adapters with you.
• Lightning talks must be anecdotal.

I would do this again if I could!!

My usual error checking function while writing Golang looks something like:

func check(err error) {
    if err != nil {
        panic(err)
    }
}

(read this rant for an idea of Go’s error management)

This is what I have cluttered around my code and until now, I have been a big fan of Golang’s error propagation. Rust just does it better with it’s ? propagation operator!

let file = File::open(filename)?;    // propagate error to caller

Exceptions in C++ are just a mess. They’re unreadable and really do not encourage fine-grained handling of errors. They’re also very goto-ish. Combined with the match keyword, the propagation operator is a powerful tool. Code looks elegant and readable as well.

I only seriously started learning Rust recently. But the journey has been enjoyable. I expect more rants to come on Rust.

Travis CI and services like it, have shown us that the steps to build/test/deploy code should also exist in the code repo. I’m gonna try to find some time to explore the possibility of having a Travis CI style service for in-house repositories. I want to mandate the existance of a Dockerfile in every repository so we can at least tell Jenkins to go do a docker build . in every repo in the list. But this is only temporary. The long-term goal is to definitely be able to hook into Jenkins CI from the Github web interface (like Travis CI in the open source world.)

Many people talk about cross-functional teams, developers getting hit by buses (poor us!) and CI/CD. I am not one to disagree here but I think these other tools are important as well as using the tools available to best effect.

Git

I am having to use Gerrit with a Gitweb frontend all maintained by a local team on servers somewhere upstairs. You want a new repo? Please write an email to this guy.. Private repository? No.

The first thing I did when I joined was to set up my own Gitlab instance. This enabled me to do the best practices again (check-in code daily, write markdown README.md files etc. etc. Yes, our repo doesn’t understand README.md) We’re now moving to Github Enterprise which is definitely awesome.

Slack

Skype for business should be banned! I have missed slack since day 1. Slack let you create channels for different stuff going on like #github and #pull-requests etc. It was a place where different people from different teams could #discuss #everything. This revolution of hashtags somehow missed out on people here. No more.

Mattermost and RocketChat are great alternatives to Slack. I set up Mattermost for my team and that’s where we plan our Kicker tournaments. (I don’t like the ~ that RocketChat uses for it’s channels.) I have not tried Zulip yet but one of my cousins works on it.. It looks really nice too.

The org however still runs on Email and Skype for business. Something needs to change! Perhaps Microsoft Teams is a good idea?

RTC

RTC stands for Rational Team Concert (Whatever That Means). Its the crappiest tool I’ve ever had to deal with but it’s mandated across the organization. It’s meant to be another tool from IBM that is your enterprise agile management system with strange compliance certifications.

I’d rather use Jira or something. I have a Trello-like server (wekan) which I use as a Kanban board. Otherwise, I’m stuck with RTC.

Github has cool project boards. I should probably start using that since we are moving to Github anyways.

CI and CD

I am a huge fan of having a Jenkinsfile, .travis.yml file or something of this sort as part of your repository. Why we don’t have this now is beyond me. This is something I’m working on evangelising:

Every repo must contain:

• Jenkinsfile (even if it’s empty!)
• Dockerfile (to at least build the app)

I’m also a big fan of having your CI/CD servers automated. Working on this, one-thing, one best-practice at a time!

Conclusion

This is all for today. But I have noticed that your enthusiasm for change is often never reciprocated. Keep chugging! I have been able to achieve some of the things I mentioned above but only after loads of frustration. I still see people having long-winded discussions on email threads though. Bad habits aren’t easily lost.

Skunkworks are healthy. I’m a proud revolutionary (for the better) from the underground. I hope I’ll look back at these frustrations and be proud :) Get people around you that are as enthusiastic for change and march forth!

Behold the vid: https://youtu.be/NK1z3095YvM

… and the code: https://github.com/shrmrf/container-demo

Although the first thing you’ll note is that I didn’t write a proper test. I just find it more pleasurable to go ahead with the code but with a specification at the top. While presenting, I was arguing where the syscall.SysProcAttr configurations could be placed and forgot to actually call the child() function. This caused me to move into the wrong direction and I was about to reel off under pressure (remember, this was a live session). But luckily I figured out what I was doing wrong (I had actually practised the code the day before, so that might’ve helped.)

Its a 7 minute video, but I think it gives you a nice idea of what goes behind-the-scenes in docker’s magic.