Why?

When using a containerized-first architecture, two issue may arise.

1. The need of having private repositories AKA private images.

2. The need of having fast image fetching for multiple servers inside a local network, even for public available images.

To solve the first problem we need to have a private registry, and to solve the second we need to have a registry to mirror the public registry currently in use.

What?

The good old Docker Hub registry is part the CNCF, and is publicly available under the name Distribution.

As of now Distribution supports one of two type of deployments, either as a mirror (proxy) or as a private registry, as noted in the official website

Pushing to a registry configured as a pull-through cache is unsupported

But, we can find a way around it.

Using NJS

Nginx is one of the most popular revers-proxy out there offering resilient, reliable and fast proxy to various type of services. One of the less-known features of Nginx is NJS or Ngnix JavaScript.

NJS is integrated inside Nginx and can be enabled with one line. After enabling NJS, Nginx abilities are extended to the full power of programming-driven logic, you can see a lot of NJS examples here.

We're going to utilize NJS for enabling two Distributions deployment act as one, offering both mirroring option and private registry option.

Inside the JS code we're going to check whether the request method is GET for mirroring and existing images, or any other method for pushing images into our private registry.

As we can see in the diagram the deployment is composed of a Nginx service standing as the entry-point to our infrastructure, separating the requests to two different Distribution deployments based on the request method.

Both of the Distribution services are mounting the same storage as their repo volume storage.

How?

We'll deploy our registry using Docker Swarm using three Docker Swarm managers - with no workers as it's not required - behind load balancer, like such:

Each one of the Docker Swarm node manager need to have access to the same Storage volume. For that, we can utilize either nfs or any S3 compatible service.

Our nginx.conf is pretty short

upstream mirror {
  server registry-mirror:5000;
}

upstream private {
  server registry-private:5000;
}


js_import mirror from conf.d/mirror.js;
js_set $mirror mirror.set;

server {
    if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" )  {
              return 404;
          }
  client_max_body_size 2G;
  listen 8080;

  location / {
    proxy_pass        http://$mirror;
    proxy_set_header  Host              $http_host;
    proxy_set_header  X-Real-IP         $remote_addr;
    proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header  X-Forwarded-Proto $scheme;
    proxy_read_timeout                  900;
  }
}

We're setting two upstreams. One for the mirror, and one for the private one. the decision to which one we'll route the traffic is done using NJS as we can see in these two lines

js_import mirror from conf.d/mirror.js;
js_set $mirror mirror.set;

The first line loads the file conf.d/mirror.js into an object named mirror, then it sets the value into a variable named $mirror using the mirror.set function. When the value is setup is being used as the proxy_pass value.

proxy_pass        http://$mirror;

Last thing left to investigate is the code mirror.js itself.

export default {set}

const mirror      = 'mirror'
const privateRepo = 'private'

function set(r) {
    let c = mirror;

    if (r.method !== 'GET')
        c = privateRepo;

    return c;
}

Here we're exporting the function set, that checks to see if the request method is not GET and if so it will set the upstream to the private upstream.

To deploy our custom-registry we need to do the following steps:

1. Create a Docker Swarm leader

2. Join another 2 nodes as leaders as well

3. [in case of nfs, attach the nfs storage to the leaders]

4. deploy the stack using the docker-compose.yml file into the Swarm.

Prior to all servers setup make sure you're placing all serves in the same local network using VPC.

Create a Docker Swarm leader

Setup a Ubuntu server, and install Docker by running the following command as a root user.

curl -fsSL https://get.docker.com | sh

Then, init the swarm by running

docker swarm init

That's it you've a working docker swarm.

Run the following command to get the command for joining another managers to the Swarm.

docker swarm token join-token manager
# docker swarm join --token Very-Long-Token 10.0.16.230:2377

Join another 2 nodes

Setup another 2 Ubuntu servers and install Docker in them, then run join command from above

docker swarm join --token Very-Long-Token 10.0.16.230:2377

Now, run this command from anyone of the managers to see all attached nodes (machines).

docker node ls

Attach storage or set S3 credentials

Now it's time to attach the storage, we can use one of two options (other options are of course available, but we'll focus on those two) nfs="network file system" or S3 compatible storage. We'll R2 which is S3 compatible storage provided by Cloudflare with free egress (bandwidth) usages

Clone the lab repo and cd into the lab folder, then choose either nfs or r2.

git clone https://github.com/byawitz/labs.git
cd labs/02-your-own-cr/docker
# For nfs
cd nfs
# For r2
cd r2

R2

To set an R2 edit the docker-compose.yml file and adjust the the S3 endpoint, replace ACCOUNT_ID with your Cloudflare account ID.

REGISTRY_STORAGE_S3_REGIONENDPOINT: https://ACCOUNT_ID.r2.cloudflarestorage.com/

Then, create admin read/write/delete token within your Cloduflare r2 settings and paste the values into files/r2-access-key.txt and files/r2-secret-key.txt accordingly. you can move to the last step "Deploy the stack".

NFS

To set an NFS storage, setup another Ubuntu server and install the nfs service by running

apt update
yes | apt install nfs-kernel-server

Then, create a folder and grant it will all permissions.

mkdir -p /mnt/nfs/registry
chmod 777 /mnt/nfs/registry/

Find your local network subnet and run the following command, replacing the 192.168.1.0/24 with your current subnet.

echo "/mnt/nfs/registry  192.168.2.0/24(rw,sync,no_subtree_check)" >> /etc/exports

This command will append a line to the file /etc/exports states everyone within that subnet will have a read-write access to that folder.

Now run these two commands to apply the exports and restart the nfs-kernal server.

exportfs -a
service nfs-kernel-server restart

Optional: in case you're using firewall you can run this command to let any machine with the local subnet accessing the nfs server.

ufw allow from 192.168.2.0/24 to any port nfs

Now for the last step, find out the nfs server internal IP by running ip a, then, run the following commands on each of the Swarm servers, replace 192.168.2.10 with your nfs local IP.

apt update
yes | apt install nfs-common
mkdir -p /nfs/registry
echo "192.168.2.10:/mnt/nfs/registry    /nfs/registry/    nfs    defaults    0 0 " >> /etc/fstab
mount -a

This will

1. Install the nfs-common that will utilize the nfs client,

2. Create a local folder /nfs/registry/

3. Add mounting point for the nfs remote directory into our newly created one.

4. will mount the folder, from now on it will happen automatically during system boot.

Deploy the stack

After setting the storage we can run the stack by running

docker stack deploy -c docker-compose.yml cri

That's it

You're custom registry is now ready and can be used as a mirror a private registry, or both. Go build something awesome!

Image by freepik

Kubernetes are the industry standard for deploying scalable applications.

Kubernetes is shortened to k8s where the 8 mark the 8 characters between the k and the s

There are two general ways to use k8s:

1. Managed: The k8s cluster is already deployed for you, and you only need to manage the applications life-cycle. Offered by many companies including Amazon, Azure, Google, DigitalOcean, Linode and much more. You can find the whole official list in the CNCF landscape in the Certified Kubernetes - Hosted section.

2. Self-Managed: In this way you'll also need to deploy and manage the cluster yourself. For that you have a variety of distributions to choose from which can be found on the same link above in the Certified Kubernetes - Distribution section.

Application developers (CKAD) mostly go with a Managed solution, while Kubernetes Administrator (CKA) will go with last one, check here for more details about k8s certificates.

k3s is one of the official k8s distributions that meant to be a light weight version of k8s.

Due to its light-weight nature k3s was named like such as it meant to be half of k8s.

k8s has 10 characters so k3s is half, 5 characters and as such the 3 marked the middle 3 characters.

There's no long abbreviation for k3s.

Why?

k8s is a full-feature solution, meant to cover all use-cases. As of such k8s is a complex product, and the task of deploying k8s cluster makes it complex as well.

k3s is a much more lighter version, and deploying a basic k3s cluster takes less then a minute. Even complex deployment of k3s don't take more the 30-90 minutes.

k3s achieves that by providing light, stripped-out version of k8s that lack many of the large-scale features. Because of that you can have a single-node cluster of k3s. And, due to it's lightweight nature it will consume small amount of resources while letting you deploy apps in k8s style.

Although k3s lack some k8s features, it's a CNCF certified distribution of k8s, meaning you can use your yaml file against k3s and it will work!

One of the biggest advantages of starting using k3s in your environments is the that you're getting use to deploy application the k8s way. This way when changing to k8s, managed or self-managed, the only thing that changes is the cluster engine, and not the way you're interacting with it.

k3s vs k8s

The main difference of "k3s is lighter the k8s" doesn't give us that much, will try to break the differences a bit more.

These features makes k3s less robust then k8s, but much more suitable for other user cases as:

• Edge computation

• Home labs

• Internal k8s clusters

• Starting point to get use to k8s

• Any scale of application that not uses k8s as of today

The last bullet-point is important, and why? leaving managed containers as ECS outside, most of today application will be deployed in one of the following approaches:

• Traditional way: which you're installing and maintain the require infrastructure as the need grows. For example, when you suddnley have a need for NodeJS in your environment, you just install it.

• Docker & Docker compose way: a single server with container management.

• Docker Swarm: Docker orchestration tools.

The first approach is hard to maintain, while the second don't scale (easily at least) which leaves us with Docker Swarm.

Docker Swarm is a good approach for HA and FT applications, though in some point as your applications grows you'll find yourself in a need of k8s.

k3s will provide you deployment options that goes from a single server up to the HA cluster that shares the capabilities of Docker Swarm. Meaning, you can have the same benefits as you have today and starting to shift towards k8s-style deployments.

Deploying

Architecture

k3s can be deployed on a bare metal, VPS and even raspberry pie, in one of two ways as a k3s Server or as a k3s Agent.

Image form k3s documentations.

k3s server includes additional tools for managing the cluster, while k3s agent includes only the necessary tools needed for running pods, services and other k8s resources, and a way to communicate with the k3s server(s), the k3s supervisor.

As can be seeing in the architecture, k3s always uses just one-binary to handle all the is components and tools.

By default k3s servers will be used as an agent for when deploying resources. We can block such behavior by tainting the k3s server nodes.

Tainting in k8s are a way to declare which resources can be deployed in a given node by declaring key value pair, for example:

kubectl taint nodes k3s-server1 taint=server:NoSchedule

This command will set the taint=server key-value pair for the k3s-server node. and any resource that needed to be able to deployed on that node will need to add toleration in their declaration for the key-value pair, for example

tolerations:
- key: "taint"
  operator: "Equal"
  value: "server"
  effect: "NoSchedule"

The NoSchedule effect declares that this taint rule are from now on and don't apply for running resources.

By default k3s comes with traefik as a built-in ingress controller. Make sure to target this ingress class in your yaml files. This feature - as almost everything in k3s - can be disabled.

We will cover the following deployments.

• One k3s server

• Three k3s servers for high availability

• One k3s server and three k3s agents.

• One k3s server with taint and three k3s agents.

• Three k3s server with taint and three k3s agents.

One Server

Will start with the basics.

In this example will deploy a single-node k3s cluster.

We can see that in this deployment both the end-users and the cluster-managers that uses the kubectl will accesse the cluster the same way.

To deploy the cluster, ssh into your server and run:

curl -sfL https://get.k3s.io | sh -s - server

The command will fetch the content of https://get.k3s.io and will execute using sh. That's it, you now have a fully working k3s cluster.

You can get your config file from /etc/rancher/k3s/config.yaml and copy it to your local machine to use it with kubectl.

Three Servers

In this deployment we will achieve high-availability by deploying 3 servers into the cluster.

We're not deploying any agents yet as the servers are not taint and can be used as agents too.

In this deployment we will put all the servers behind load-balancer.

In is default settings k3s uses sqlite as is data-store, that won't work anymore as our cluster is now distributed. For distributed clusters k3s supports either using distributed etcd or using an external SQL db, will go with distributed etcd.

When using etcd the servers IP must be a local one, meaning all servers must share VPC and placed in the region. for cross-region cluster deployment you'll to need use the external db option.

ssh into the first server and run

curl -sfL https://get.k3s.io | sh -s - server --cluster-init

Using the --cluster-init flag will install k3s with etcd. (you can use this command also to convert existing single-node k3s from using sqlite to use etcd).

Now we need to have the k3s token. The token location is /var/lib/rancher/k3s/server/token and we can get as such,

$ cat /var/lib/rancher/k3s/server/token
K107df354fe86b....::server:THIS_IS_THE_TOKEN

The last part after the ::server: is the token, we will use that token and the existing server internal IP to join other servers as such:

  curl -sfL https://get.k3s.io | sh -s - server \
    --server https://10.0.0.3:6443 \
    --token "THIS_IS_THE_TOKEN"

Now you can check all the nodes in the cluster as such:

:~# kubectl get nodes
NAME    STATUS   ROLES                       AGE     VERSION
k3s-1   Ready    control-plane,etcd,master   2m18s   v1.29.3+k3s1
k3s-3   Ready    control-plane,etcd,master   96s     v1.29.3+k3s1
k3s-2   Ready    control-plane,etcd,master   2m52s   v1.29.3+k3s1

You can see that each of the nodes has the control-plane and etcd roles.
To access the cluster run your kubectl against the load balancer IP.

One Server three Agents

In this deployment we will deploy one server node using the default sqlite data-store and will connect 3 agents.

Because we haven't taint the server node it will be included in the load balancer target for the end-users.

In the server node will run

curl -sfL https://get.k3s.io | sh -s - server

That will create a single-node k3s cluster, now we can connect agents using the the server token and IP, as such:

  curl -sfL https://get.k3s.io | sh -s - agent \
    --server https://10.0.0.3:6443 \
    --token "THIS_IS_THE_TOKEN"

Run this command in the server or in your local machine using kubectl

:~# kubectl get nodes
NAME    STATUS   ROLES                  AGE     VERSION
k3s-1   Ready    control-plane,master   3m22s   v1.29.3+k3s1
k3s-3   Ready    <none>                 18s     v1.29.3+k3s1
k3s-2   Ready    <none>                 11s     v1.29.3+k3s1
k3s-4   Ready    <none>                 57s     v1.29.3+k3s1

One Tainted Server three Agents

This deployment is the same as the one above, the only different is that we are going to taint the server node.

As you can see the load balancer is not pointing to server node anymore.

Run this command to taint the server node. (replace k3s-1 with your server node name.

kubectl taint nodes k3s-1 manager=only:NoExecute

Now we can check the node taints as such:

$#: kubectl describe nodes k3s-1 | grep Taint
Taints:             manager=only:NoExecute

This will prevent placing any resources on the server node.

TheNoExecuteeffect means that current services will get evicted from the node.

Three Servers (tainted) three Agents

This would be the most complex deployment of them all.

In this deployment will have high-availability for the server nodes which are all going to be tainted. All of the server nodes will be behind a load-balancer which will be used by the agent nodes and the cluster manager.

First we run this to deploy single-node k3s cluster with etcd enabled.

curl -sfL https://get.k3s.io | sh -s - server --cluster-init

In Hetzner for example you can create a server without external IP whatsoever which would be very useful for the server nodes.

Now, we run this on the other servers.

  curl -sfL https://get.k3s.io | sh -s - server \
    --server https://10.0.0.2:6443 \
    --token "THIS_IS_THE_TOKEN"

Now we can taint all of our servers by running

kubectl taint nodes k3s-server-1 k3s-server-2 k3s-server-3 manager=only:NoExecute

And we can add agents using the servers load balancer IP address

  curl -sfL https://get.k3s.io | sh -s - agent \
    --server https://10.0.0.6:6443 \
    --token "THIS_IS_THE_TOKEN"

Now we just need to add another load balancer in-front of the agents.

Tips to go

To uninstall k3s from your server you'll have a ready to use file inside /usr/local/bin

# For server nodes
/usr/local/bin/k3s-uninstall.sh

# For agent nodes
/usr/local/bin/k3s-agent-uninstall.sh

It's recommend to first delete the node before uninstalling it, for example

# In a server node
$server: kubectl delete nodes agent-12

# In the agent node
$agent-12: /usr/local/bin/k3s-agent-uninstall.sh

To have a better grasp on your cluster practice how:

• Add servers and agents.

• Set alarms for when a node is down.

• Backup and restore the etcd or sqlite cluster database.

Finally, use a firewall in-front of your cluster, specially in front of your server nodes.

Image by pikisuperstar on Freepik

We'll use the following tools for creating a self-hosted analytics system.

The requirements for this article are to have general knowledge in:

• Docker & Docker Swarm

• Nginx

• and, Stream processing in general.

The Idea

When utilizing existing tools together we'll be able to have a production ready product. And, all what we need to do is to just update the underlying technologies being in used.

Here's the end results 👇

Why?

The main advantage of self-hosting solutions is to have control over the data.

The second, it's usually cheaper.

Last advantage, this type of production environment scaled easily, meaning, you need a wider throughput? just add more nodes containing the same services and put it all behind a load balancer.

Architecture

First, will use nginx as the only direct endpoint to the server. nginx will reverse proxy each request based on the domain to one of two inner endpoints:

1. benthos for event capturing

2. grafana for accessing the analytics dashboard

benthos will have http_server input configured to handle incoming requests. Then, benthos will check to see if the message is indeed for the app, and will add some meta data, then it will deliver (produce) the data directly to a kafka topic, and response with 201 (empty successful response) to the user immediately.

The reason for that is to make sure the event capturing process will fast as possible, between 10-50ms total! sending the event to kafka is a good way to ensure just that.

On the other hand we have a vector container subscribed to the kafak topic for receiving the events. vector will process each event and will add them to OLAP database.

In the process vector will parse the user-agent, the user country and other event details.

Then, the management-user can access the grafana dashboard to view data in real-time or to filter by dates.

Deployment Steps

In the lab repo you can find terraform files for one-liner deployment of N+B+K+V+G+C stack to Hetzner and Cloudflare (for DNS).

Files and Folders

Clone the lab repo and cd into 01-your-own-analytics/docker. You will see these files.

• docker-compose.yml The main architecture file

• .env Contains all necessary environment variables as Database passwords, etc.

• nginx/analytics.conf Contains nginx config file for analytics endpoint with rate-limit of 60 requests per second.

• nginx/dashboard.conf Contains nginx config file for Grafana dashboard endpoint.

• bentohs/config.yml - Benthos config file

• vector/config.yml - Vector config file

• vector/mmdb - MindMap GeoLite database.

• grfana/dashboards/*.json dashboards settings

• grfana/provisioning/dashboards/clickhouse.yaml Dashboards loading instructions.

• grfana/provisioning/datasources/clickhouse.xml ClickhHouse datasource settings.

• scripts/install.ts Installer script for creating ClickHouse database and table.

• (Optional) scripts/seed.ts Seeding a year-worth of dummy data.

We use docker-compose.yml file to describe the architecture for ease of use, and it would be suitable for most use-cases. in large environment you'll probably use Docker Swarm or a fully blown Kubernetes environments.

The project contains a justfile which its a simple script runner or "just is a handy way to save and run project-specific commands."

You can download the right one for your system here.

For local development with just you can run, you'll be ask for password, this is for editing your hosts file, check inside the justfile for more details.

just install-local

To remove everything, run

just remove

Now, let's go through the deployment process.

Setting nginx endpoints

First, we need to update the domains inside nginx/analytics.conf and nginx/dashboard.conf files. Here's the analytics configuration for example.

limit_req_zone $binary_remote_addr zone=base_limit:10m rate=5r/s;

server {
  ...
  server_name analytics.test;

location / {
    limit_req zone=base_limit;
    ...
    proxy_pass http://input-pipeline:4195;
   }
}

You can also notice that we are setting rate-limit for the analytics endpoint only, as we want to prevent abuse against the endpoint from which we are gathering data.

The rate limit settings are, 5 Requests per second for given IP.

As for the dashboard endpoint it's recommended putting it behind Firewall or a different type of blocker as CloudFlare ZeroTrust which is free up to 50 users.

Env file

Inside the .env file you can set strong password and different usernames for the databases. The only field you might want to change is PROJECTS

PROJECTS=app,website

This variable contains list of projects names. This project names will be in use when you will create events.

Installation

Now, we can start everything by running

docker compose up -d

Creating Kafka topic.

Run this command to execute binary inside the kafka container to create Kafka topic named analytics with 7 days message retention, after that Kafka will delete them.

docker exec event-store kafka-topics.sh \
--create --topic analytics \
--config retention.ms=604800000 \
--bootstrap-server localhost:9092

Now it's better to restart the vector container as the topic didn't exist till now.

docker compose restart vector

Creating Clickhouse table

Run this command to create Clickhouse table.

export $(grep -v '^#' .env | xargs) \
&& docker run --rm  \
--env CLICKHOUSE_HOST --env CLICKHOUSE_USER --env CLICKHOUSE_PASSWORD \
--env CLICKHOUSE_PROTOCOL --env CLICKHOUSE_PORT --env CLICKHOUSE_DB \
--network=analytics \
-v ./scripts:/app oven/bun bun /app/install.ts

This command will run the typescript file using bun runtime.

Optional - seed some demo data

To see the dashboard with data while browsing you can run the following command to seed years worth of data into the database.

export $(grep -v '^#' .env | xargs) \
&& docker run --rm  \
--env CLICKHOUSE_HOST --env CLICKHOUSE_USER --env CLICKHOUSE_PASSWORD \
--env CLICKHOUSE_PROTOCOL --env CLICKHOUSE_PORT --env CLICKHOUSE_DB \
--network=analytics \
-v ./scripts:/app oven/bun bun /app/seed.ts

Create your first event(s)

Now we are ready to create our first event. we'll use curl to POST an event to our Benthos (through nginx) endpoint.

curl --location 'http://analytics.test' \
--header 'X-App: app' \
--header 'Content-Type: application/json' \
--data '{
    "type": "click",
    "user_id": "1",
    "session_id": "FaAa31",
    "timestamp": "2024-04-17T04:04:12",
    "locale": "en",
    "path": "/getting-started",
    "appVersion": "0.0.1",
    "details": {
        "meta_a": "meta_a",
        "meta_b": "meta_b"
    }
}'

or JavaScript fetch for example

const body = JSON.stringify({
  "type": "click",
  "user_id": "1",
  "session_id": "FaAa31",
  "timestamp": "2024-04-17T04:04:12",
  "locale": "en",
  "path": "/getting-started",
  "appVersion": "0.0.1",
  "details": {
    "meta_a": "meta_a",
    "meta_b": "meta_b"
  }
});

await fetch("http://analytics.test", {
  method: "POST",
  headers: {
    "X-App": "app",
    "Content-Type": "application/json"
  },
  body,
});

User ID vs Session ID

User ID is an identifier that you'll try to maintain as long as you can as this is the way you can different between users, for mobile you can use some device id like identifierForVendor in iOS, and AndroidID for Android. In web, you can store random unique ID inside the browser Local Storage.

Session ID on the other hand is only for short period of time, for that you can create a random unique id combined with the current user ID and store it in memory. In the browser you can also store it inside Session Storage.

View the data in Grafana

Now access your Grafana. For the first time you'll to login using admin as username and password. Then, you'll be prompted to change the password.

You can also see events occurring in real-time by enabling Grafana auto load from the upper top right.

That's it!

ClickHouse is very! efficient, 119,716,577 rows weight total of 21.91GB, but, on disk it's compressed and weight only 4.93GB!

You can deploy a N+B+K+V+G+C stack on a € 16.18 server at Hetzner and get

• 4 vCPU

• 8GB RAM

• 160GB SSD

• and 20TB! Bandwidth,

To have that number of events you'll need to create 9,976,381 events a month, or 332,546 events a day, or 3-4 events in a given second.

This deployment of N+B+K+V+G+C its just simple example, and, you can of course adapt it to your needs.

🪄 Expelliarmus!

Background Image by bunny on Freepik

Stream processing

Will start with answer to the question What is Stream processing at all?

Event-Driven-Development

To address that question, we need to take a step back and examine traditional development versus event-driven development.

In the traditional approach, also known as request-response development, the flow is linear and occurs synchronously. For instance, consider a transaction on an e-commerce website. The steps would be as follows:

1. The user enters credit card details and submits the form.

2. The details are sent to the backend.

3. The backend charges the card via the payment gateway while the user waits.

4. After successfully charging the card, the backend updates the database.

5. The backend sends an invoice to the user.

6. The backend sends an email to the warehouse to initiate order fulfillment.

7. Finally, the backend sends a response indicating success.

In many cases, the traditional approach suffices; however, there are instances where it can be considerably error-prone.

Firstly, the flow can diverge or alter at any point if one of the steps fails. For instance, suppose the payment charge and invoice sending succeed, but the warehouse notification fails. How do you track and recover from such failures?

Additionally, the end-user must wait throughout the entire process, whether it takes 2 seconds or 60. While this may be tolerable for many scenarios, it can pose a significant issue in others.

For instance, consider IoT applications where each device must report to a central base every N seconds. If the base's response time exceeds a few milliseconds, it can lead to a bottleneck of data, blocking other IoT device requests and compromising overall throughput.

To summarize, the issues with the request-response approach are:

1. Lack of fault tolerance and inability to recover from failures.

2. Blocked processing until the entire process is completed.

To address these challenges and handle a large volume of requests, you may consider adopting Event-Driven Development (EDD).

In EDD, the flow operates asynchronously. The key components of EDD are:

1. Event store: Such as Kafka or RabitMQ, where all events are stored

2. Producer: Responsible for sending events to be stored in the event store, organized by topics

3. Consumer: Registers in the event store to receive all events from specific topic(s).

For instance, a producer might generate an event in the "function-execution" topic, containing execution data. Subsequently, consumers subscribed to the "function-execution" topic will receive the event data and process it accordingly.

This approach redefines the flow of the e-commerce example within an EDD framework.

1. The user enters credit card details and submits the form.

2. The details are sent to the backend.

3. The backend produce a charge event and stores it in the event store.

4. The backend sends a success message from the event store (the user receives a response).

5. The charging consumer retrieves the event from the event store, charges the card, and sends the event back to the event store.

6. Then, the invoice consumer processes the event, sends it back, and continues the flow.

7. Then, the warehouse consumer processes the event and sends it back.

8. Then, the invoice consumer sends the event back to the end-user..

In this example, Event-Driven Development (EDD) may appear more complex than the traditional approach. However, by introducing one more step, we gain several significant benefits:

• Fast: Producing events takes mere milliseconds, ensuring that your throughput remains unblocked.

• Asynchronously: Event processing within consumers occurs in the background, reducing the processing load on the main endpoint.

• Fault tolerant: Most data stores retain all events on disk, making the system resilient to failures.

• Guaranteed delivery: Each event must be committed, ensuring that events are resent until processing is completed successfully.

• Extendible: Adding additional steps is straightforward; just add a new consumer.

• Scalable and distributed: Event stores like Kafka are distributed by default, making them easy to scale across multiple servers and consumers.

Stream processing

When it comes to handling messages from an event store like Kafka, stream processing plays a crucial role. Stream processing can be achieved using various SDKs or by utilizing a stream processor.

A stream processor is a pre-built solution that typically consists of three main components:

1. Inputs or Sources - These are the origins from which the streams obtain their data. Examples include Kafka, AWS SQS, URL endpoints, and system metrics.

2. Enrichment or Processors - Messages received from the inputs are processed here. Examples of processing tasks include filtering, removing sensitive data, type conversion, validation, and more.

3. Outputs or Sinks - This is where the processed data is sent. It could be a database, an S3 bucket, or even another Kafka instance.

Why choose a stream processor over writing code?

While it's possible to achieve similar results by writing custom code, stream processors offer several advantages. One key advantage is that stream processors come with everything pre-configured, and they often cover most common use cases. This means you can get up and running quickly without needing to set up and fine-tune every aspect of your data processing pipeline.

Stateless vs Stateful streaming

In stream processing, there are two main approaches: stateless and stateful processing, each defining how incoming data is handled.

Stateless

In stateless processing, events are processed as they arrive without considering previous event processing. This means each event in the stream undergoes the same processing steps and follows the same pipeline.

Common examples of stateless streams include:

• Metrics gathering

• Database replication

• Log aggregation

• General tasks that don't rely on knowledge of other events.

ile it's possible to process data based on past events in a stateless approach, it typically requires manual modifications and isn't inherent to the process.

Stateful

In the stateful approach, the processing pipeline can dynamically adjust based on previous data. Examples of stateful processing include:

• Infrastructure abuse monitoring: Requires past event data for effective monitoring, as gathering this information without historical context can be challenging.

• Fraud detection

• Search result optimization

• Any event data that benefits from insights gained from previous events.

Stateful processing is well-suited for scenarios where historical context or insights are crucial for accurate and meaningful data processing.

Both Vector and Benthos are stateless stream processors. While Vector does offer some stateful options, such as RAM-State, these are temporary and will be lost if Vector is terminated.

Why about others?

Other popular solutions for stream-processing are:

Maybe some day we'll do an article to explain why so many packages for the same thing.

Each of those solution is a bit different then the others, but the main reasons to use Benthos or Vector is because they are language agnostic, both of them uses yaml with toml in Vector.

Conceptual differences

Idea

The main difference between Vector and Benthos lies in their primary objectives:

Vector aims to serve as a data/log/metrics aggregator, employing a stream-processing approach.

Benthos, on the other hand, focuses on providing a stream-processing solution specifically for events.

Despite these differing goals, both solutions are capable of handling many stream processing tasks, which is why a comparison between them is relevant and useful.

Lang

Vector is written in Rust, and Benthos is written in Go. Both are a very efficient general-purpose languages. And, even that some might claim that Rust is faster we will omit that advantage from Vector.

Usage differences

Both Vector and Benthos provide a diverse range of inputs, processors, and outputs. To compare their differences effectively, we'll focus on components that are unique to each solution.

Inputs - Sources

The differences in the inputs is that Vector is more toward logs, and as of such it includes log inputs like:

• Docker logs

• k8s logs

• Datadog agent and more

Benthos goes towards the events goal and haves inputs as:

• SQL Select (for mysql, postgres, clickhouse, mssql, sqlite, oracle, snowflake, trino,and gocosmos)

• Discord (new message)

• and, Twitter search.

Both of the solutions shared support for the most popular services as:

• AMQP

• AWS Kinesis Firehose

• AWS S3

• AWS SQS

• HTTP Client - Check for update

• HTTP Server - Listen for update.

• File

• Kafka

• Redis

• Socket

• GCP PubSub

• Exec (getting output from a local binary)

• stdin

Vector Remap Language (VRL) vs Bloblang

Voth VRL in Vector and Bloblang in Benthos serve as type-agnostic languages for remapping and processing various data formats like JSON, XML, or others. They allow you to transform and manipulate data without being tied to a specific data type, which is particularly useful in handling diverse event data.

Given this event data

{
  "events": [
    "1:file_uploaded",
    "2:alram_set_up",
    "3:file_found",
    "4:sytsem_shutdown"
  ]
}

As we can see the events data contains the : to separate between the event ID and the event data, and the event data space uses _. we'll use both languages to enrich that event data. We'll start with Bloblang

root = this.events.map_each(event -> {
  "id": event.split(":").index(0),
  "data": event.split(":").index(1).split("_").join(" ")
})

This code will map each of the events to an id and data object and will put the results inside the root element, so will have something like such:

[
  { "data": "file uploaded", "id": "1" },
  { "data": "alram set up", "id": "2" },
  { "data": "file found", "id": "3" },
  { "data": "sytsem shutdown", "id": "4" }
]

TIP: Run benthos blobl server to get fully working REPL for practicing your Bloblang

In Vector VRL is a bit different. (Use this website to practice on VRL)

e = []

for_each(array!(.events)) -> |_index, value| {
    a, err = split(value,":")
    text, err2 = split(a[1], "_")
    joined, err3 = join(text, " ")
    e= append(e,[{"id":a[0], "event":joined}])
}

. = e

The main difference is that VRL variables don't have methods so instead of text.split we must write split(text), imo it gives Bloblang a small advantage.

Output - Sinks differences

In the outputs section we once more see the goal diffrences, while both of the solutions supports common outputs as:

• AWS Kinesis Data Firehose logs

• AWS Kinesis Streams logs

• AWS S3

• AWS SNS

• AWS SQS

• Azure Blob Storage Console

• GCP Cloud Storage

• GCP PubSub

• HTTP

• Kafka

• Redis

Vector will go more toward the logs way as loki and Benthos towards the general data organizer as sftp.

Scalability

Scalability is a key advantage of stateless tools like Benthos and Vector. They can scale easily by adding more instances of the tool. Running them in Docker containers is a recommended approach for ensuring smooth operation without interference.

For large deployments, Benthos offers features like a Kubernetes (k8s) operator and a Helm chart for managing deployments efficiently. Similarly, Vector also provides a Helm chart for streamlined deployment.

One unique advantage of Vector is its agent-to-aggregator concept. This allows you to distribute workload among Vector instances and centralize stream processing, creating a single source of truth. For instance, each server can have a Vector agent gathering Docker logs, which are then sent to another Vector acting as an aggregator, responsible for filtering and enriching the data.

Extending

Want to create your own inputs, processors or outputs? for now it's possible only with Benthos. check this repo.

Example

Both of the tools can be run in the same way.

$# benthos -c config.yml
$# vector --config config.yml

The config.yml will contain all description of the pipeline process.

Benthos

input:
  generate:
    mapping: |
      root.id = uuid_v4()
      root.name = fake("name")
      root.email = fake("email")
    interval: 1s

pipeline:
  processors:
    - mapping: |
        root.id = this.id
        root.length = this.id.length()
        root.details = this.name + " - "+this.email
    - mapping: root = this.format_xml()
output:
  broker:
      outputs:
        - stdout: { }
        - file:
            path: ./a.txt
            codec: lines

In this example we are running the generate input every 1 second, and generating an object with fake ID, fake name and fake email.

Then, we use the mapping processor. First adjusting the field, and, then formatting the output as XML.

Then, the data will be output to the stdout and will be appended to a local file named a.txt.

Vector

sources:
  generate_syslog:
    type: "demo_logs"
    format: "json"
    count: 100

transforms:
  remap_syslog:
    inputs:
      - "generate_syslog"
    type: "remap"
    source: |
      j = parse_json!(.message)
      date = parse_timestamp(j.datetime,format: "%v %R %:z") ?? now()
      dt = to_unix_timestamp(date) / 3600
      . = {"method":j.method, "host":j.host, "referer":j.referer,"date":to_int(floor(dt))}

  remove_date:
    inputs:
      - remap_syslog
    type: "remap"
    source: |
      del(.date)
      .referer, err = replace(.referer, "https://", "")
      indexOfSlash = find(.referer, "/")
      .referer, err = slice(.referer, 0,indexOfSlash)


sinks:
  emit_syslog:
    inputs:
      - "remove_date"
    type: "console"
    encoding:
      codec: "json"

In this example will generate 100 lines of JSON demo_logs, then, in the transformers sections we first remapping the data, then removing the https and suffix part of each domain.

At the end the data being output to the console using json codec.

Conclusion

A great solution could be mixing both tools. For example, gathering metrics using Vector and send it to Benthos and so on.

If you're main goal is to gather logs and insights from your infrastructure Vector would be the way to go. In any other case using Benthos or both of the tools together will be the best.

Image by starline on Freepik

1. Introduction - you’re here

2. Basic replication

3. Multiple masters

4. Sharding - Proxy

5. NoSQL CassandraDB/ScyllaDB

A database server is typically a remote endpoint that listens for new read or write requests, which can be made through different methods such as REST, native libraries, and others.

In typical database cluster setups, there is a single source of truth, a server that contains the data and handles read and write requests.

To ensure high availability in database deployment, you should have a minimum of three sources of truth so one can take over in the event of a failure of the others.

Terminology

When discussing databases at scale, it's essential to familiarize ourselves with some new terminology, which we’ll get to know in this part:

Replication

Databases are central to managing data, and storing all your data in one place can be risky. This is where replication comes into play.

Replication serves as a secondary source of truth that maintains identical information as the original source.

Standby Server

Any data written to the main DB server is replicated to the Standby server either synchronously or asynchronously.

The Standby server remains inactive. When and if the main DB server crashes the Standby server takes over and becomes the main DB server.

When the DB servers come back it becomes the new Standby server.

Master / Slave

A Master database server listens for incoming read and write requests, while a Slave database server listens for read requests only.

Any new data written to the master is replicated to the slaves using one of two methods:

• Synchronously - data is committed to the master only when it can be inserted into all instances simultaneously, ensuring consistency at the expense of slower writing times.

• Asynchronously - data is inserted into the master on the spot, with the update happening in the background. This approach will speed up writing times but with the cost of inconsistency.

Multiple Masters

In this topology, each server also listens to write requests, unlike the previous one.

Any write request will be propagated to the rest of the servers synchronously or asynchronously.

The main advantage of this topology is that any endpoint can be used to write new data to the database, offering flexibility and convenience.

Sharding

All of the previously mentioned solutions provide redundancy but fail to address a crucial issue.

What happens if your data exceeds the capacity of one server? Even with multiple slaves or masters, each containing the same data, you can only scale your database's writing and reading capabilities, not its storage capacity.

To overcome this limitation, you can implement sharding.

When sharding, each new row is directed to a specific predetermined database server based on certain criteria.

In the example topology mentioned above, each database server contains identical tables with the same columns. For instance, when inserting an ID that starts with az, the data will be inserted into the first database server. Similarly, when reading an ID starting with ar, all the data will be fetched from the last database server.

Proxy

Except for the standby servers, you’ll need a method to access all of the servers as if they were one giant unit.

In master-master topology when all servers handle both requests a simple Level 4 load balancer can evenly distribute requests.

For master-slave and sharding topologies, you’ll need to have a proxy, You can set it programmatically using code or through a service proxy.

Code-level

For example, you can encapsulate your database connector with a code wrapper that directs requests to the appropriate database server.

In master-slave topology, you can place all the slaves behind a Level 4 load balancer and separate the request by reading or writing, for example, this pseudo code.

const masterIP = '10.0.0.15';
const loadBalancerIP = '10.0.0.60';

actionToServer(isWrite = false) {
  let ip = loadBalancerIP;

  if(isWrite) {
     ip = masterIP;
  }

  return ip;
}

In a sharding topology, you would check the ID of the request to map it to the correct server. Here's an example:

const sharding = {
  "aa": "192.168.1.4",
  "ab": "192.168.1.5",
  "ac": "192.168.1.6",
  "ad": "192.168.1.7",
  "az": "192.168.1.8"
}

function shardKeyToServer(id = 'aa') {
  return sharding[id];
}

Service-proxy

Using a code-level proxy can be beneficial but may be more challenging to maintain. For this reason, a service proxy can be a useful alternative.

Two popular service proxies are:

• ShardingSphere

• ProxySQL

Mixing

It’s common to use a mix of the above topologies, for example:

1. The first circle can be sharding topology.

2. Then, each sharding is deployed using a master-master topology.

3. Then, each of those masters has master-slave topology

NoSQL vs YesSQL

While the approaches mentioned above require significant effort, they are typically associated with Relational Databases (YesSQL), as NoSQL databases often come with these functionalities built-in.

The reason is due to YesSQL's main advantage, joins.

In YesSQL you don’t write any data twice - at least you should avoid it - instead, you’re creating foreign keys to reference data from other tables.

However, a challenge arises when you scale your database to multiple servers. For instance, if you want to join data from server 1 to server 2, each request to the database will multiply, especially with complex joins.

To address this issue in YesSQL databases, strict sharding is essential. This involves ensuring that all inserted data, including foreign key values, resides on the same physical server. However, achieving this strict sharding setup is not always achievable.

Going forward

In the upcoming sections of the series, we will delve into the fundamentals of deploying these kinds of topologies.

To make all the examples easy to understand in the coming articles we’ll use docker and docker-compose.