Not for prod but here’s a resolution:

Install Metrics Server

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.6.4/components.yaml

Apply the patch:

kubectl -n kube-system patch deployment metrics-server --type=json \
-p='[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--kubelet-insecure-tls"}]]'

thanks to @awalker125

(Credits for the original list @beshrkayali)

I have started to add to a blacklist here:

I also created a regex which I’m experimenting with: ^r[0-9]{1,2}-{3}sn-h0je[a-z0-9]*.googlevideo.com

I tested this RegEx on regex101.com with the following data:

r4---sn-h0jeener.googlevideo.com
r4---sn-h0jeener.googlevideo.com
r4---sn-h0jeln7l.googlevideo.com
r1---sn-h0jelne7.googlevideo.com
r2---sn-h0jeln7e.googlevideo.com
r1---sn-h0jeln7e.googlevideo.com
r3---sn-h0jeenle.googlevideo.com
r1---sn-h0jeenle.googlevideo.com
r5---sn-h0jeened.googlevideo.com
r1---sn-h0jeenek.googlevideo.com
r5---sn-h0jeln7l.googlevideo.com
r2---sn-h0jeenle.googlevideo.com
r1---sn-h0jeln7e.googlevideo.com
r5---sn-h0jeen76.googlevideo.com
r5---sn-h0jeln7y.googlevideo.com
r5---sn-h0jeenek.googlevideo.com
r5---sn-h0jeened.googlevideo.com
r5---sn-h0jelne7.googlevideo.com
r1---sn-h0jeener.googlevideo.com
r4---sn-h0jeln7l.googlevideo.com
r1---sn-h0jelne7.googlevideo.com
r4---sn-h0jeln7r.googlevideo.com
r2---sn-h0jeenek.googlevideo.com
r6---sn-h0jeen76.googlevideo.com
r3---sn-h0jeener.googlevideo.com
r1---sn-h0jelne7.googlevideo.com

DISCLAIMER: THE FOLLOWING CAN LEAD TO CREATING SECURITY HOLES IN THE VPN-PROTECTED NETWORK… BE VERY CAREFUL. MAKE SURE YOU KNOW WHAT YOU’RE DOING. IF YOU DON’T UNDERSTAND WHAT YOU’RE DOING, DO NOT PROCEED

VPNs are a pain in the neck. I managed to find a way to have VPN traffic limited to some sites instead of routing everything through your corporate network.

Routing everything through corporate network is also a cause of load on the company’s servers. Imagine a ton of people watching Youtube over VPN. Now, I’m talking about traditional VPN here, not p2p stuff.

Use vpn-slice

vpn-slice is an awesome tool that allows you to use the VPN for only a certain few websites. I used vpn-slice along with openconnect.

Installation is pretty straightforward if you follow the documentation. Just make sure the root user sees the binary. I installed pyenv seperately for the root user and configured the global python env to be 3.9.2 (pyenv global 3.9.2).

That’s it. Now install vpn-slice:

$ pyenv global 3.9.2  # or whatever version you want
$ pip install --upgrade pip
$ pip install dnspython  # recommended in project README
$ pip install vpn-slice

Usage with openconnect

# Source: https://docs.microsoft.com/en-us/microsoft-365/enterprise/urls-and-ip-address-ranges?view=o365-worldwide#skype-for-business-online-and-microsoft-teams
MSFT="teams.microsoft.com login.microsoftonline.com microsoftstreams.com 13.107.64.0/18, 52.112.0.0/14, 52.120.0.0/14"
MYCOMPANY=""  # Your company's domains, IP addr etc.

openconnect \
  -c /home/$USER/.cert/CERT.crt \
  -k /home/$USER/.cert/private.key --key-password-from-fsid \
  -s "vpn-slice $MSFT $MYCOMPANY" \
  vpn.company.com

Terminology

Bare-Metal is a a physical computer and it’s base OS

Host is a bare-metal machine that runs VMs

Thread is a core of a CPU

CPU thread refers to the thread on the host system CPU

vCPU thread is a thread on a VM’s virtualized CPU

Note: I borrowed terminology heavily from this article on null-src

Isolating CPUs

My last article covers this topic. Isolating a VM’s vCPU threads from the host system ensures that host’s processes don’t interfere with the VM.

Scheduling Openstack VMs on isolated cores

CPU Affinity

using taskset

Linux Process Scheduler Tuning

GRUB cmdline

Okay… don’t delete the console entries in your command line… the entries below are for isolated cpu 1. This is not a comprehensive list… Just append whatever makes sense to you.

GRUB_CMDLINE_LINUX_DEFAULT="threadirqs rcu_nocbs=1 rcu_nocb_poll isolcpus=1 nohz_full=1 nohz=off intel_pstate=disable nosoftlockup nohalt"

Isolating CPU Cores

In a StarlingX cluster CPU isolation and Kernel commandline parameters are (generally) managed from the active controller through system host-cpu-XXX commands.

For isolation, we have the Application-isolated assigned_function.

Get current functions:

$ system host-cpu-list controller-0
+--------------------------------------+----------+-----------+----------+--------+-----------------------------------------+-------------------+
| uuid                                 | log_core | processor | phy_core | thread | processor_model                         | assigned_function |
+--------------------------------------+----------+-----------+----------+--------+-----------------------------------------+-------------------+
| 65b8337e-08bf-4e32-b7e6-17505eaf0806 | 0        | 0         | 0        | 0      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Platform          |
| b7578ba2-cba0-46cd-b860-3ba5c03ba91d | 1        | 0         | 1        | 0      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Platform          |
| ff7447a2-00df-43db-8716-d1416faf89bc | 2        | 0         | 2        | 0      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 142796e5-b17d-48e9-b18a-b84c1ea5ae37 | 3        | 0         | 3        | 0      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| c68a8dc7-367f-4512-84ca-ec559fbb63f4 | 4        | 0         | 4        | 0      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 15f360e2-d27c-4d9a-b0ff-0b78387eeb8c | 5        | 0         | 5        | 0      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 89274cb5-4fb5-4f5b-a8c5-ec36394b5ce5 | 6        | 0         | 6        | 0      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 33fb1e8e-77cd-4334-b061-dd40e6b7f7ba | 7        | 0         | 7        | 0      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 8c804a31-c61e-408c-83c4-3c7192a861d8 | 8        | 0         | 0        | 1      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Platform          |
| 560ad09d-48df-42ec-8588-7e739030d7e3 | 9        | 0         | 1        | 1      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Platform          |
| cbd83713-0118-4995-9810-486cfdbb912e | 10       | 0         | 2        | 1      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| cb45713a-fa51-4628-b536-e3d5d57c891e | 11       | 0         | 3        | 1      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 28dff3de-9913-4055-89d5-09dfc084c3c1 | 12       | 0         | 4        | 1      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 98cf6c9e-2717-4f88-b487-487dbdadd18e | 13       | 0         | 5        | 1      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 73dcec10-7a62-4d07-ba23-3482d0e0f2af | 14       | 0         | 6        | 1      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
| 4a5a4477-f0db-4d77-9a6a-c77e5ca292e4 | 15       | 0         | 7        | 1      | Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz | Application       |
+--------------------------------------+----------+-----------+----------+--------+-----------------------------------------+-------------------+

The system host-cpu-modify command is used to modify the assigned function.

$ system host-cpu-modify
usage: system host-cpu-modify [--nowrap] -f <function>
                              [-p0 <num_cores_on_processor0>]
                              [-p1 <num_cores_on_processor1>]
                              [-p2 <num_cores_on_processor2>]
                              [-p3 <num_cores_on_processor3>]
                              <hostnameorid>

For example, to set the 15th logical core to Application-isolated:

$ system host-cpu-modify -f Application-isolated -p0 15 controller-1

stx-openstack

One application that can make use of core isolation is stx-openstack. For example, if there are realtime VMs that require scheduling on isolated/dedicated cores, we need to set the compute key in the helm overrides (see OpenStack docs for: cpu_dedicated_set and cpu_shared_set).

Create a file for nova overrides that looks like this:

conf:
  overrides:
    nova_compute:
      hosts:
      - conf:
          # DELETED FOR CONCISENESS
        name: controller-0
      - conf:
          # DELETED FOR CONCISENESS
        name: controller-1
      - conf:
          nova:
            DEFAULT:
              # NO CHANGE
            compute:
              cpu_dedicated_set: '"1-2"'
              cpu_shared_set: '"3-5"'
            libvirt:
              # NO CHANGE
            pci:
              passthrough_whitelist:
                # NO CHANGE
            vnc:
              # NO CHANGE
        name: worker

Then, run the following command to apply these overrides:

$ # Update Helm overrides
$ system helm-override-update --values <yaml-file-with-overrides>.yaml stx-openstack nova openstack
$
$ # Apply application!!
$ system application-apply stx-openstack

Now, Openstack is able to schedule VMs on isolated CPU cores.

OpenStack Flavor

The following properties should be added to the RT VM (hugepages not yet included)

$ openstack flavor show rt.medium
+----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                      | Value                                                                                                                                                           |
+----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                                                                                                                                           |
| OS-FLV-EXT-DATA:ephemeral  | 0                                                                                                                                                               |
| access_project_ids         | None                                                                                                                                                            |
| disk                       | 10                                                                                                                                                              |
| id                         | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx                                                                                                                            |
| name                       | rt.medium                                                                                                                                                       |
| os-flavor-access:is_public | True                                                                                                                                                            |
| properties                 | hw:cpu_policy='dedicated', hw:cpu_realtime='yes', hw:cpu_realtime_mask='^0', hw:cpu_thread_policy='isolate', hw:emulator_threads_policy='share', hw:pmu='False' |
| ram                        | 2048                                                                                                                                                            |
| rxtx_factor                | 1.0                                                                                                                                                             |
| swap                       |                                                                                                                                                                 |
| vcpus                      | 2                                                                                                                                                               |
+----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+

Note on 2 vcpus: one will be used by linux and the other will be utilized by the RT program (cyclictest for instance)… You may use any number of vcpus … just make sure that one vcpu is dedicated for linux.

• ESC: I have remapped the Escape key to CAPSLOCK. Who uses CapsLock? ESC on the other hand is a Vim user’s go-to reflex. So keep it close-by.
• ESC+HJKL: For navigation. Why would you use muscles to move over to the arrow keys when you have them under your fingers.

How-to

xmodmap can be used to modify these key settings. But I think it will only work for X-sessions.

I found dconf/org/gnome/desktop/input-sources/xkb-options to be a much more solid solution.

Under xkb-options you can put a “Custom value” like: ['caps:swapescape'] (note: syntax is important!!)

Windows Note

By far the best and most reliable key remapper on Windows is sharpkeys (randyrants/sharpkeys).

• KubeEdge
• k3s and k3os
• StarlingX

I present my brief impressions of each below. This post will need updates as my research goes on.

Background

Edge is often used as a vague term for either one of CDN, factory network or IT data center. There is a fair amount of confusion where cloud transforms into “Edge”.

Fog apparently is used to describe the same concepts and maintains the vaporous water analogue that is so popular in this space (think about it: data lake, fog, clouds …) Usage of the term “Fog” has been tainted by Cisco. They make a poor attempt to show the difference. Meh.

kubernetes is a container orchestrator that has become extremely popular in the last few years. It provides a common API to manage all containers running in your system.

Cloud native is used to describe container-first and container-based systems. (my own definition)

CNCF is the cloud native computing foundation. It manages projects that specifically target cloud native systems. It does not manage all cloud-native projects. To the contrary, there is a small list of projects that are managed by CNCF.

Linux Foundation maintains a set of open source projects. LF Edge is the subgroup that does this for projects targeting Edge deployments. LF projects are of all types.

OpenStack Foundation is another foundation that was originally formed for OpenStack’s projects (next item) but it seems to now be accepting projects related to other systems.

OpenStack is the most popular open source cloud. It’s the OG thing: was popular before docker, k8s or anything container-related was mainstream. There are still many users of OpenStack in the wild and it’s not going anywhere soon.

KubeEdge

KubeEdge is a CNCF sandbox project designed to extend Kubernetes from cloud to edge. The IoT Edge has unique challenges that no one solution can provide a complete solution for. KubeEdge is designed to meet the challenge of:

• Network Reliability
• Resource constraint; and
• Scalability

KubeEdge has agents running on the Edge. The control plane resides on the Cloud and if there’s a network failure/disconnect, the agents keep on doing the necessary control operations. Metadata is synchronized as soon as the connection is restored.

KubeEdge is an opinionated take on the Edge: it provides services and frameworks for messaging and networking. And, since it’s opinionated, everyone might not agree with it.

My opinion is that KubeEdge will keep on serving a large userbase but it might not be extremely well suited to the 20% market that wants extra flexibility. How it pans out long-term is yet to be seen.

KubeEdge is bringing edge-edge communication which will be interesting combined with newer enhancements in the Linux kernel (read: Wireguard.) This is an extremely interesting project but for the Industry I’m working in, might not be the first choice - atleast for me.

k3s

K3s is a take on Kubernetes that removes lot’s of stuff off of standard kubernetes.

k3os

benefits of not maintaining your OS and it’s downsides

StarlingX

StarlingX is an integration project centered around Kubernetes. StarlingX makes an opinionated set of decisions for people looking to use Kubernetes for their infrastructure. StarlingX (STX) is part of the OpenStack foundation which is a testament to it’s high quality.

A simple example:

fn main() {
    let list = vec![100, 34, 72, 55];               // list is created
    let first_two = &list[0..2];                    // reference is created (first_two)
    println!("first two are {:?}", first_two);      // first_two is referred
    println!("list is {:?}", list);                 // list is referred
    println!("again, first two: {:?}", first_two);  // first_two is referred
}                                                   // first_two is dropped, then, list is dropped

A complicated (yet valid) example:

fn return_first_two(borrowed_list: &[i32]) -> &[i32] {      // borrowed_list is a reference
    &borrowed_list[0..2]
}                                                           // borrowed_list is not dropped because it references something outside of its scope

fn main() {
    let list = vec![100, 34, 72, 55];               // list is created
    let first_two = return_first_two(&list);        // reference is returned (first_two)
    println!("first two are {:?}", first_two);      // first_two is referred
    println!("list is {:?}", list);                 // list is referred
    println!("again, first two: {:?}", first_two);  // first_two is referred
}

The main takeaways from the above two examples is:

• the lifetime of values should outlive any of it’s references
• a reference’s lifetime must be completely contained within the lifetime of the value being referenced

Generic Lifetimes:

the following won’t compile

extern crate rand;
fn simulate_game(home: &str, away: &str) -> &str {
    if rand::random() {
        home
    } else {
        away
    }
}

Rust can not know the lifetime of the return value. It knows that return is valid when both references are valid!!

Adding a generic lifetime param:

extern crate rand;
fn simulate_game<'a>(home: &'a str, away: &'a str) -> &'a str {
    if rand::random() {
        home
    } else {
        away
    }
}

This syntax expresses intent in signature

Lifetime Elision:

Lifetimes are a part of the language but the language can create some common lifetimes so we don’t have to specify them all the time in our programs. This is called Lifetime Elision. The three rules of elided lifetimes:

• Each parameter gets its own lifetime
• One lifetime in parameters => return
• Self reference’s lifetime => return

If you’re not familiar with WebAssembly, I would suggest you start here:

• A cartoon intro to WebAssembly
• WebAssembly Concepts

This blog post is about wondering what WebAssembly can look like for some usecases.

Run Everywhere

WebAssembly can run everywhere - atleast that’s the goal. It’s performant - as performant as C/C++ and it’s secure.

WebAssembly is a type of code that can be run in modern web browsers and provides new features and major gains in performance.

https://developer.mozilla.org/en-US/docs/WebAssembly/Concepts

Open Questions

Although designed for the web, WebAssembly is able to run portably across architectures. This can be a nice way to distribute applications. Some day perhaps WebAssembly runtimes will be a part of all OSes.

What needs to be determined IMO is:

• Realtime capabilities

  Can WebAssembly modules written in C/C++ provide any RT capabilities? If so, then it’ll be interesting to evaluate whether a WebAssembly runtime is all that a PLC should be. Apps running the PLC would be delivered like software. Less stickiness for everybody.

• Recompile all the things

  Can WebAssembly runtimes replace native programs? How about message queues, brokers implemented in WebAssembly? I know it’s not possible to compile all complicated software to WebAssembly today - but is that the future too?

• UI

  Will Slack and it’s likes re-implement significant portions of their code to WebAssembly? Should work, right?

• Edge

  Fastly has already done lots of work with their CDN running WebAssembly-isolated workloads. Can we bring this down to the Edge where we’re running containers today?

Conclusion

There’re so many things I’m looking an answer for. This year is going to be awesome!

$ sudo apt install python3 python3-pip    # installs pip3
$ sudo pip3 install --upgrade pip         # upgrade pip to latest version

Congrats, now you’re stuck with whatever version ships with the distro. I think we can do better!

pyenv

My recommendation now, is to use pyenv with which you can manage multiple versions, virtualenvs through a nice CLI

Uninstall pip/pip3

Before proceeding, I’d suggest uninstalling pip using pip uninstall pip and sudo apt remove python3-pip

Installation

Using the pyenv installer project

$ sudo apt install libssl-dev \
                   libsqlite-dev \
                   libreadline-dev \
                   libbz2-dev  \
                   libsqlite3-dev  # I've observed failures without these libs
$ curl https://pyenv.run | bash
...
WARNING: seems you still have not added 'pyenv' to the load path.

# Load pyenv automatically by adding
# the following to ~/.bashrc:

export PATH="/home/taimoor/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

This should set up pyenv for you. Read the instructions printed at the end and follow them! Add the three things to your ~/.bashrc or ~/.zshrc or whatever.

Usage (summarized)

Example: Install Python 3.8.1:

$ pyenv install 3.8.1

P.S: logout and log back in

P.S2: Tab completion works!

Virtualenv

Create a virtualenv

$ pyenv virtualenv 3.8.1 venv3.8    # create a virtualenv called venv3.8 based on python 3.8.1
$ pyenv activate venv3.8    # activate venv3.8
$ python --version
Python 3.8.1
$ which python
/home/taimoor/.pyenv/shims/python

That’s it! Now you can manage your anaconda and python environments, versions sooo much better.

2020 used to be hindsight but not any more. It’s here. I’ll write down my own resolutions for this year here…

Technology

• Continue with my FP journey (Clojure and F#)
• Keep on learning Rust and Golang
• (Maybe) look into Ziglang
• DDD and Software Architecture. I have been reading a lot about it recently and will continue to try to do so. Especially Wlachsinn’s book!

Health

• Start exercising regularly
• Eat healthy food normally (it’s fine to eat unhealthy sometimes)
• Swim!

Misc

• Start a side-project in Rust (embedded??)
• Start a web Frontend (Clojurescript??) project

Conclusion

I’m sure I’ll miss out on some things and do some right… Fingers crossed.

I was part of a team that presented the concept of OT/IT consolidation. I got quite a few comments and questions about the demo video (in Urdu here). So here’s a shot at answering some of the questions I got forwarded:

The demo was (in oversimplified terms) showing PLC compute moved to software which can be orchestrated using Kubernetes.

Q/A

• Nice move. But how sensor data is being processed without PLC or a microcontroller?

There are “dumb” IO devices connected over Profinet in this case (but it can really be any fieldbus e.g. Ethercat etc.)

Some people had a comment about ADCs being involved. YES, but we need to think at a higher level in this case. Analog and Digital IOs are obviously involved under-the-hood.

• So what is difference between an arduino microcontroller and program able logical controller

This is a strange question. Arduino is not a microcontroller, it’s a prototyping platform (and a company as well.) This prototyping platform has a microcontroller. A PLC is a digital computer made for the industry. Follows many industrial IEC standards. It is a part of many things. For our demo, we were using ADAM PLCs from Advantech.

Arduino can also be programmed to become a PLC (see the OpenPLC project: https://www.openplcproject.com/)

• Ok good step so what precautions u take if ur factory text file or source code get hack??

If you get hacked, “precautions” are irrelevant. The word precaution means that it ought to prevent the hack. I’m assuming this was the question as well. Although this was not a security demo, but k8s brings with it good security practices (certs, https etc.). “Hack” is too open-ended of a word.

Security practices are a whole different topic. Perhaps not relevant for the discussion here.

• So can we apply machine learning and Ai codes in PLC??

I would read my answer above and conclude: NO. PLCs are meant for other things, Google is your friend!

• We get machine learning codes in Python mostly some standard c language so what u suggest??

Great attempt at moving off-topic! (Always one person in the team!!)

• Using structure text and block diagrams and ladder diagrams??

Yup

• One more thing can u tell me PLC have a capability of processing i mean controller and processor r 2 different things so can u guide me ?

They are computers in the end! The scope of their work is a bit limited to their application. Check out the OpenPLC project I linked to above.

• No. That is why I am designing machines and control systems using hybrid model. Means using PLCs, Microcontrollers and Computers to accomplish all kind of customization which include AI and custom GUIs in python and VueJs

Interesting… But we should not convolute so many things in one sentence. Everything you mentioned has its place. And they work together. We ought to avoid the use of the words “Hybrid Model” which sounds like formalism.

• Recently I have designed Pakistan’s first industrial Artificially intelligent sorting robotic conveyor using this hybrid model.

Tall claims!! I saw the video and one comment for now is that it might not meet some safety standards we have in the EU and US. Best of luck my friend!! Bring us laurels.

Also, I would tone down on Artificial intelligence and use specifics when talking to more technical folk (e.g. Computer Vision, OpenVINO, Inference, Learning etc.)

• So its coslty???

Yes, it’s supposed to run for many years so, there’s a lot of hardening.

Conclusion

I’m surprised by the interest people showed in the project. It’s still a concept and we want to work with the Industry leaders to bring about more mature products.

Getting started

Clojurescript is written in Clojure which requires Java. So we need to get all three up and running.

Java

To get Java JDK and JRE, run:

$ sudo apt install openjdk-11-jre
$ sudo apt install openjdk-11-jdk

For proxy settings, edit /etc/java-11-openjdk/net.properties.

Example for http proxy in the file:

http.proxyHost="proxy-addr.company.com"
http.proxyPort=8080
http.nonProxyHosts=localhost|127.*|[::1]

Maven

This was the hardest to figure out. And it’s crazy (coming from a newbie perspective) that maven doesn’t make it easy to just reuse Java or System proxies or respect the http_proxy variable. Madness. Moreover, you need to create the configuration file at ~/.m2/settings.xml with the following contents:

<settings>
<proxies>
    <proxy>
        <id>httpproxy</id>
        <active>true</active>
        <protocol>http</protocol>
        <host>proxy-addr.company.com</host>
        <port>8080</port>
        <nonProxyHosts>local.net|some.host.com</nonProxyHosts>
    </proxy>
    <proxy>
        <id>httpsproxy</id>
        <active>true</active>
        <protocol>https</protocol>
        <host>proxy-addr.company.com</host>
        <port>8080</port>
        <nonProxyHosts>local.net|some.host.com</nonProxyHosts>
    </proxy>
</proxies>
</settings>

Clojure

make sure rlwrap is installed (sudo apt install rlwrap)

Download and install clojure using the following commands:

$ curl -O https://download.clojure.org/install/linux-install-1.10.1.469.sh  # This is the current version as of writing
$ chmod +x linux-install-1.10.1.469.sh
$ sudo ./linux-install-1.10.1.469.sh

Using the command clj launches the repl. Thankfully!

Leiningen

Follow the steps on https://leiningen.org/

curl -O "https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein"
chmod a+x lein
mv lein ~/bin/

Create a Test Project

To test lein and figwheel+reagent:

$ lein new figwheel some-name -- --reagent

• Big ball of mud
• Monoliths
• Event-driven architectures
• Service-oriented architectures (SOA)
• Serverless

So I got to play with the Data Plane Development Kit (DPDK) from Intel recently. I planned on replicating the PVP benchmark for my use-case. I set up the following environment:

• Ubuntu 18.04 (bionic) host with:
  • KVM
  • QEMU
• Debian 10 guest images downloaded from: https://cdimage.debian.org/cdimage/openstack/current/

A useful tool is numactl (use apt to install it). On my non-server system, I don’t have multiple socks and multiple cores per sock etc.

root@machine:~# numactl  -s
policy: default
preferred node: current
physcpubind: 2 3 4 5 6 7 
cpubind: 0 
nodebind: 0 
membind: 0 

Setting up (OVS+DPDK)

First of all set install OVS-dpdk using apt

sudo apt-get install openvswitch-switch-dpdk 
sudo update-alternatives --set ovs-vswitchd /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk

Restart the systemd service: sudo systemctl restart openvswitch-switch.service

Hugepages will need to be set up. But we need to tell DPDK about that.

root@machine:~# cat /etc/dpdk/dpdk.conf
   ...
NR_1G_PAGES=4
   ...

Isolate some CPUs

Edit the value of GRUB_CMDLINE_LINUX_DEFAULT to:

GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=4 hugepagesz=2M hugepages=2048 iommu=pt intel_iommu=on isolcpus=0,1"

And run sudo update-grub.

this setting will not need to change afterwards. A reboot of the machine is required. See if the parameters got passed correctly by doing cat /proc/cmdline

Set up Hugepages

sudo mkdir -p /mnt/huge
sudo mkdir -p /mnt/huge_2mb
sudo mount -t hugetlbfs none /mnt/huge
sudo mount -t hugetlbfs none /mnt/huge_2mb -o pagesize=2MB
sudo mount -t hugetlbfs none /dev/hugepages

root@machine:~# grep HugePages_ /proc/meminfo 
HugePages_Total:       4
HugePages_Free:        1
HugePages_Rsvd:        0
HugePages_Surp:        0

Configure OVS-DPDK

The database needs to be set up only once (sudo ovs-vsctl --no-wait init)

sudo ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0x03  # hopefully: cores, 0,1 
sudo ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="1024"
sudo ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x0C # we have just one NUMA node

Create OVS-DPDK Bridge and Ports

dpdkvhostuser is a predefined type. vhost-user is understood by Qemu 2.2 and above.

sudo ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
sudo ovs-vsctl add-port br0 vhost-user1 -- set Interface vhost-user1 type=dpdkvhostuser
sudo ovs-vsctl add-port br0 vhost-user2 -- set Interface vhost-user2 type=dpdkvhostuser

root@machine:~# ovs-vsctl show
c3459670-39b8-43f7-bc57-569672ed79c6
    Bridge "br0"
        Port "br0"
            Interface "br0"
                type: internal
        Port "vhost-user2"
            Interface "vhost-user2"
                type: dpdkvhostuser
        Port "vhost-user1"
            Interface "vhost-user1"
                type: dpdkvhostuser
    ovs_version: "2.9.2"

Binding Devices to DPDK

We should check whether the NIC is compatible with DPDK first. The list on DPDK doesn’t seem to have been updated e.g. the documentation itself for newer versions states that the I210 is supported.

sudo modprobe vfio-pci
sudo dpdk-devbind --bind=vfio-pci eno1 # eno1 is the interface I want to bind to DPDK

dpdk-devbind --status can be used to check whether the NIC is now using the DPDK compatible driver.

root@machine:~# dpdk-devbind --status

Network devices using DPDK-compatible driver
============================================
0000:00:19.0 'Ethernet Connection I217-V 153b' drv=vfio-pci unused=e1000e

Network devices using kernel driver
===================================
0000:02:00.0 'Killer E220x Gigabit Ethernet Controller e091' if=enp2s0 drv=alx unused=vfio-pci *Active*

    ...

VMs Quick Start

Download Debian 10 (buster) and change the password for root using guestfish. (Debian was an arbitrary choice here.)

Add anything to the images at this point. I downloaded qperf with all it’s dependencies and added it:

PACKAGES="qperf iperf"
apt-get download $(apt-cache depends --recurse --no-recommends --no-suggests \
  --no-conflicts --no-breaks --no-replaces --no-enhances \
  --no-pre-depends ${PACKAGES} | grep "^\w")

virt-copy-in -a ./debian-10.0.0-openstack-amd64-2.qcow2  *.deb  /home/

openssl passwd -1 <password> can be used to generate a password entry, and this can be added to the guest image using:

guestfish --rw --add ../debian-10.0.0-openstack-amd64.qcow2 --mount /dev/sda1:/ vi /etc/shadow

And make 2 copies of the debian-10.0.1-20190708-openstack-amd64.qcow2 file. We’ll use it as our hard drives for guests.

Some resources to read

• File-sharing: https://www.linux-kvm.org/page/9p_virtio
• OVS+KVM: https://docs.paloaltonetworks.com/vm-series/8-1/vm-series-deployment/set-up-the-vm-series-firewall-on-kvm/performance-tuning-of-the-vm-series-for-kvm/enable-open-vswitch-on-kvm.html#
• Using testpmd to test DPDK Performance: https://software.intel.com/en-us/articles/testing-dpdk-performance-and-features-with-testpmd
• Vhost/Virtio in DPDK: https://software.intel.com/en-us/articles/configuration-and-performance-of-vhost-virtio-in-data-plane-development-kit-dpdk

Launching the Guests

I launched the guests in two separate terminal windows (i.e., copy-paste instead of executing).

root@machine:~# cat ./launch-os.sh 
# Launch OS1
qemu-system-x86_64 -m 1024 -smp 4 -cpu host -hda ./debian-10.0.0-openstack-amd64.qcow2 -boot c -enable-kvm -no-reboot -net none -nographic -chardev socket,id=char1,path=/run/openvswitch/vhost-user1 -netdev type=vhost-user,id=mynet1,chardev=char1,vhostforce -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1 -object memory-backend-file,id=mem,size=1G,mem-path=/dev/hugepages,share=on -numa node,memdev=mem -mem-prealloc

# Launch OS2
qemu-system-x86_64 -m 1024 -smp 4 -cpu host -hda ./debian-10.0.0-openstack-amd64-2.qcow2 -boot c -enable-kvm -no-reboot -net none -nographic -chardev socket,id=char2,path=/run/openvswitch/vhost-user2 -netdev type=vhost-user,id=mynet2,chardev=char2,vhostforce -device virtio-net-pci,mac=00:00:00:00:00:02,netdev=mynet2 -object memory-backend-file,id=mem,size=1G,mem-path=/dev/hugepages,share=on -numa node,memdev=mem -mem-prealloc -fsdev local,security_model=passthrough,id=fsdev0,path=/tmp/share -device virtio-9p-pci,id=fs0,fsdev=fsdev0,mount_tag=hostshare

Godspeed!

Using annotations, you can:

• Customize the Kernel that is used (see KernelPath example below)
• Change the guest image (ImagePath)
• Hypervisor (HypervisorPath)
• Firmware (FirmwarePath)
• etc. etc. (All of the annotations are listed in annotations.go)

Using a Custom Kernel

Here is the yaml file I used to apply a custom kernel:

apiVersion: v1
kind: Pod
metadata:
    annotations:
        com.github.containers.virtcontainers.KernelPath: "/usr/share/kata-containers/vmlinuz-4.19.31-40"
    name: kata-example
spec:
  runtimeClassName: kata-qemu

  containers:
  - name: nginx-container
    image: nginx

The metadata key contains the annotations map. All of the annotations are listed in annotations.go on the main repo.

This is documented here: https://github.com/kata-containers/packaging/blob/master/kata-deploy/README.md#docker-quick-start

Using Minikube and Kata

This PR is adding the documentation to get started with minikube: https://github.com/kata-containers/documentation/pull/445#

Using kubeadm

• First, set up a Kubernetes cluster using kubeadm: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
• then, https://github.com/kata-containers/packaging/blob/master/kata-deploy/README.md#kubernetes-quick-start

Conclusion

Using the new kata-deploy method, it is very easy to set up a Kubernetes cluster that uses Kata Containers.

For now, let’s look at how I went about doing what I did..

1. You need to go get github.com/kata-containers/packaging repository. (or, git clone if you so prefer.) Be sure to have your $GOPATH properly set!!
2. cd into packaging/kernel directory.
3. I chose the 4.19.31 version of the kernel. Primarily because it was a newer kernel (not too different from the kernel version that comes with the current kata release.) and because it had the RT patches available
4. I changed the kernel_version="" line to kernel_version="4.19.31"

That’s it. We are ready to (1) apply the necessary patches from Kata maintainers to our Kernel of choosing and (2) apply the RT patches:

# download and apply the kata patches to the kernel
./build-kernel.sh setup

cd kata-linux-4.19.31-36
patch -p1<../path-to-extracted-patch/patch-4.19.31-rt18.patch

Wait… we need to first, make menuconfig and enable RT capabilities

General Setup 
  --> Preemption Model
    --> Fully Preemptible Kernel (RT)

Now, we’re ready to build the Kernel and install it*:

./build-kernel.sh setup
sudo ./build-kernel.sh install

Drum Roll please

$ docker run -it --runtime kata-runtime ubuntu bash
root@77cf04fd8290:/# uname -r
4.19.31-rt18
root@77cf04fd8290:/# uname -a
Linux 77cf04fd8290 4.19.31-rt18 #2 SMP PREEMPT RT Mon Apr 29 09:06:08 CEST 2019 x86_64 x86_64 x86_64 GNU/Linux
root@77cf04fd8290:/# 

So proud!!!

Happy Building!!

Note on Kernel Files

vmlinux: uncompressed kernel. | vmlinuz: compressed kernel. | bzImage: located under arch/x86_64/boot/bzImage. Big Zimage. Compressed kernel image.

More info: https://stackoverflow.com/a/22338835/3760442

Acknowledgements

Special thanks to @egernst, @rico and the great team on Slack!! Though this was my first foray into Kata Containers, I must say the folks were very welcoming. Thank you all!!

* I built the kernel (as of the time of writing) using make build -j4 instead of using the build-kernel.sh script but it shouldn’t matter. Just FYI.

Conclusion

What I concluded from the documentations is that we don’t get something ready-to-go from the official docs. And you should consider yourself lucky if everything worked for you out-of-the-box.

Setup Ideas

• Start from a clean installation of Ubuntu 18.04.
• The local.conf goes into the top-level directory of the devstack checkout.

Setup

My setup already provided the DNS and DHCP servers as illustrated below.

        (DNS Server)              (DHCP Server)
             |                          |
             |                          |
             +------------+-------------+
                          |
                          |
                        (OpenStack)

So, my local.conf looks like the following:

[[local|localrc]]
ADMIN_PASSWORD=safeandsecure
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD

HOST_IP=<HOST_IP>

FLOATING_RANGE=<HOST_IP>/24
Q_FLOATING_ALLOCATION_POOL=start=<HOST_IP>.150,end=<HOST_IP>.200
FIXED_RANGE=10.11.12.0/24
FIXED_NETWORK_SIZE=256
PUBLIC_INTERFACE=eno1
PUBLIC_NETWORK_GATEWAY=<HOST_IP>.1

LOGFILE=$DEST/logs/stack.sh.log
LOGDAYS=2

SWIFT_HASH=66a3d6b56c1f479c8b4e70ab5c2000f5
SWIFT_REPLICAS=1
SWIFT_DATA_DIR=$DEST/data

(The <HOST_IP>.XXX numbers are obviously replacing the last part of the IP address.)

What I’m telling OpenStack is that I …

• already have a Gateway in the network, use it!
• Already know what the IP allocation pool looks like

Next

With that, hopefully you can go do OpenStack stuff and not have to worry about networking being the biggest issue in your life. Good luck!!!

Benefits

• The “environment” you’re programming against are the function arguments
• Reuse is imminent. Composition is what everyone calls it.
• Functions’ results can be cached (memoization)

There must be many more benefits.. These three are the ones I can think of and the ones that get the most marks from my side.

Main Concepts

Functional programming is a technique that depends on the programmer adhering to some principles (it helps if the programming language forces him as well):

Purity

A pure function has no side effects i.e. it depends only on its arguments.

(defn pure-function [input]
  (if input output-1 output-2))

In this example, if the pure-function is really pure, then depending on the input, the return value always remains the same. This also means that the output-1 and output-2 data are immutable i.e. they’re unchangable!

Referential Transparency

In the example above, we might just have replaced the output of pure-funtion with either output-1 or output-2 thus avoiding the call to a function. Many languages can optimize this behavior by caching the result. It’s called memoization. This also means that you could parallelize stuff i.e. functions can be independently evaluated on different machines/cores/nodes whatevers.

Persistence

Immutability is key. You never mutate. An operation on data doesn’t change the data, it just returns new data. This naive approach is memory hungry I suppose but platforms and languages find ways to optimize stuff.

Laziness

Since I’m learning Clojure, there might be a bias because seq are everywhere and they’re laaaaaaaaaaazy! That is, you only evaluate stuff when it’s required and never before. This enables infinite length sequences. Also, we pay for operations only when required.

Conclusion

I’m still a Functional Programming newbie but Clojure is opening my eyes! I am intrigued.

Fin!

• Learn Clojure: I want to learn some FP this year
• Improve Rust: work on some internal/external tools at work
• TSN: important in industrial applications
• Yocto: Finally master this crap
• ClearLinux: What I want to be working on
• Firecracker: Get this thing running on my hardware

Non Technical Goals

• Exercise: get a routine in place
• Read: Some book every month
• Manage: manage time better

This list is kept to a minimum so the goals are achievable. Wish me luck!!

Lightning Talk

I did a lightning talk around Event Driven Microservices. Event driven architectures are becoming fairly common these days and IoT is a natural and easily understandable use-case. I presented the architecture we developed but without going into too much detail.

One question I had some trouble answering was around the “infinite-length” event streams. What if the queues get filled up.. How do we save state? The answers to these questions is it depends most of the time but usually you don’t run out of space at all :). And I refer to Martin Fowler for these questions.

Demo

I did an “Fearless Systems Programming with Rust” tutorial/demo as well. It was all going well… I presented a PowerPoint slide deck with some introductory stuff. And then “let me switch to a REAL OPERATING SYSTEM” said I. And I realized that I had a 13" Dell XPS. D’oh!! It only had a Thunderbolt connection and it wouldn’t be able to connect to the projector.

Then, I braved up.. Went unprepared into my Lab machine somewhere else and starting setting up a new project that did my container-demo style version of containers in Rust. Basically, going through the basics of containers/docker and writing those in Rust. (Thanks again Liz!)

Because I was now doing live coding without a prepared machine over ssh across Germany (lag and all ..), the demo didn’t go too smooth! But in the end I was proud that we did get the final product right and in the process we did (literally) demonstrate the power of the Rust compiler, and the Cargo toolchain.

Conclusions

• Take all the adapters with you.
• Lightning talks must be anecdotal.

I would do this again if I could!!

The goal is to introduce events and derivatives of events. Why events are the source of truth for all software. Below I dump my thoughts so I can come back to them later.

Final Example

(11-11-2018) Today, discussing with a friend, I think I came up with the case-study I’m gonna go with.

Advantages of Microservices

Explain the advantages of microservices.

• Programming language independence
• Scale the bottlenecks

Advantages of Event Triggered Architectures

Explain the advantages of event-triggered systems.

• Fire and forget type of architecture
• IoT is like that
  • Publish/Subscribe
  • Any component can pick up the data it needs

Final story

I’ll start with a brief overview of Microservices and event triggered systems.

• A recommendation engine with a broken algorithm.
• Events are used to recommend the next video to play.
• The algo is messed up and recommends a certain medicine to all the male viewers OR there’s an update that we think the customers will be more happy with.
• In an event triggered system
  • deploy new recommendation service
  • events get replayed in the new service and better recommendations come up

It is natural that to get your first k8s cluster running you’d want to remove the taint from the master node and use it as the cluster (i.e. a single-node cluster). The first thing that a person would do is not set the kubelet network plugin (–network-plugin=cni). It’s certainly what I did wrong.

The problem is you don’t see the problem until later because the orchestrator will use docker’s default networking which would quickly fail in a more-than-one node cluster. So, it’s best practice to immediately apply a pod network!!

Device Sharing

What if you want to orchestrate the usage of resources (like GPU, audio, crypto) using Kubernetes? Its best practice to create a device plugin. NVIDIA has a k8s-device-plugin for it’s GPUs. If this plugin isn’t used, you basically end up having to install NVIDIA drivers inside the docker container and hope the host’s version of the driver is exactly the same and share the devices with the container.

This is more elegant. Do it!

• runc is the CLI tool for running containers (create/run/delete/manage)
  • docker uses runc as its runtime.
  • Kubernetes uses runc as its runtime through cri-o (those “unsupported Docker version detected” errors you’ve probably seen.. yeah!) There’s no guarantee that runc will remain cri-o compatible.
• cri-o is an interface for container runtimes. Any compatible runtime can be used. runc is one of those others being rkt (never tested it), Kata containers and I guess many more.
• containerd - runtime manager created by docker and later contributed to CNCF. It’s the daemon that’s running in the background doing your bidding that you pass through runc.

Coding interviews are supposed to test the programming abilities of the testee. Although coding interviews ideally test the ability of the test taker to program, they are far from perfect.

What do I classify as coding interviews? I’d say interviews that involve the test-taker to do any sort of programming is a coding interview. I’d classify whiteboard interviews in the same category.

What are the goals?

• I’d say the main goal is to weed out the worst CVs that claim years of experience in a programming language and the person/applicant is not really a good programmer.

• Another goal is to test if the applicant is versed in the basic computer science concepts of performance tradeoffs etc. i.e. O(someting) PLUS the space-time tradeoffs.

Where coding interviews fail

• Online coding challenges fail to catch cheats. I could sit with a friend and have him do solution 4 while I work on 1.
• The interviewer may be biased: not really ever obvious, but it’s humans doing interviews in the end.
• In-person interviews might make the applicant confused. This is just a matter of experience. I think experience helps here.

Conclusion

I like the concept of in-person whiteboard sessions. They are abstract enough so that the applicant can feel comfortable with his knownledge, he also doesn’t need to remember syntax by heart.

Development environments and editors are a personal thing for developers and programming in a foreign environment isn’t the same as doing it on your own machine. Recently, I had to talk to a board over serial (tty) interfaces on a colleague’s machine. Took 2 hours of struggle to get nothing done in the end. Went downstairs to my machine, sshd into the board and 5 min(!) later, done! 2 hours!! I’m not joking.

My usual error checking function while writing Golang looks something like:

func check(err error) {
    if err != nil {
        panic(err)
    }
}

(read this rant for an idea of Go’s error management)

This is what I have cluttered around my code and until now, I have been a big fan of Golang’s error propagation. Rust just does it better with it’s ? propagation operator!

let file = File::open(filename)?;    // propagate error to caller

Exceptions in C++ are just a mess. They’re unreadable and really do not encourage fine-grained handling of errors. They’re also very goto-ish. Combined with the match keyword, the propagation operator is a powerful tool. Code looks elegant and readable as well.

I only seriously started learning Rust recently. But the journey has been enjoyable. I expect more rants to come on Rust.

Travis CI and services like it, have shown us that the steps to build/test/deploy code should also exist in the code repo. I’m gonna try to find some time to explore the possibility of having a Travis CI style service for in-house repositories. I want to mandate the existance of a Dockerfile in every repository so we can at least tell Jenkins to go do a docker build . in every repo in the list. But this is only temporary. The long-term goal is to definitely be able to hook into Jenkins CI from the Github web interface (like Travis CI in the open source world.)

Many people talk about cross-functional teams, developers getting hit by buses (poor us!) and CI/CD. I am not one to disagree here but I think these other tools are important as well as using the tools available to best effect.

Git

I am having to use Gerrit with a Gitweb frontend all maintained by a local team on servers somewhere upstairs. You want a new repo? Please write an email to this guy.. Private repository? No.

The first thing I did when I joined was to set up my own Gitlab instance. This enabled me to do the best practices again (check-in code daily, write markdown README.md files etc. etc. Yes, our repo doesn’t understand README.md) We’re now moving to Github Enterprise which is definitely awesome.

Slack

Skype for business should be banned! I have missed slack since day 1. Slack let you create channels for different stuff going on like #github and #pull-requests etc. It was a place where different people from different teams could #discuss #everything. This revolution of hashtags somehow missed out on people here. No more.

Mattermost and RocketChat are great alternatives to Slack. I set up Mattermost for my team and that’s where we plan our Kicker tournaments. (I don’t like the ~ that RocketChat uses for it’s channels.) I have not tried Zulip yet but one of my cousins works on it.. It looks really nice too.

The org however still runs on Email and Skype for business. Something needs to change! Perhaps Microsoft Teams is a good idea?

RTC

RTC stands for Rational Team Concert (Whatever That Means). Its the crappiest tool I’ve ever had to deal with but it’s mandated across the organization. It’s meant to be another tool from IBM that is your enterprise agile management system with strange compliance certifications.

I’d rather use Jira or something. I have a Trello-like server (wekan) which I use as a Kanban board. Otherwise, I’m stuck with RTC.

Github has cool project boards. I should probably start using that since we are moving to Github anyways.

CI and CD

I am a huge fan of having a Jenkinsfile, .travis.yml file or something of this sort as part of your repository. Why we don’t have this now is beyond me. This is something I’m working on evangelising:

Every repo must contain:

• Jenkinsfile (even if it’s empty!)
• Dockerfile (to at least build the app)

I’m also a big fan of having your CI/CD servers automated. Working on this, one-thing, one best-practice at a time!

Conclusion

This is all for today. But I have noticed that your enthusiasm for change is often never reciprocated. Keep chugging! I have been able to achieve some of the things I mentioned above but only after loads of frustration. I still see people having long-winded discussions on email threads though. Bad habits aren’t easily lost.

Skunkworks are healthy. I’m a proud revolutionary (for the better) from the underground. I hope I’ll look back at these frustrations and be proud :) Get people around you that are as enthusiastic for change and march forth!

Behold the vid: https://youtu.be/NK1z3095YvM

… and the code: https://github.com/shrmrf/container-demo

Although the first thing you’ll note is that I didn’t write a proper test. I just find it more pleasurable to go ahead with the code but with a specification at the top. While presenting, I was arguing where the syscall.SysProcAttr configurations could be placed and forgot to actually call the child() function. This caused me to move into the wrong direction and I was about to reel off under pressure (remember, this was a live session). But luckily I figured out what I was doing wrong (I had actually practised the code the day before, so that might’ve helped.)

Its a 7 minute video, but I think it gives you a nice idea of what goes behind-the-scenes in docker’s magic.

You’ll find my CV here soon. (Todo: Add link, I haven’t added a link because right now, there’s too much info on the CV i.e. phone number etc.)

I love to work, hack and have fun with software. Rust is my new favorite thing.

This is why the Flux architecture intrigues me: a restriction (one-way data flow) liberates you from a world of spaghetti. The coder is forced (or at least, strongly tempted) to create abstractions, dispatchers, stores and queues so that he can meet this criteria. Uni-directional dataflow also has the advantage of inherently suggesting the developer that he should make abstractions based on functionality.

Embedded systems can also benefit from flux designs: events generate information and dispatchers are responsible for broadcast events. This should make embedded systems more modular in design and possible to be extended. I look forward to experimenting with flux-designs in backends or embedded systems in the days ahead.