Page Comparison

Exploratory approach with minikube

Clean the lab (lab work)

sudo iptables -F
sudo iptables -F -t nat
sudo minikube delete
sudo docker system prune -a
sudo reboot

...

alias minikube-kill = `docker rm $(docker kill $(docker ps -a --filter="name=k8s_" --format="{{.ID}}"))`
alias minikube-stop = `docker stop $(docker ps -a --filter="name=k8s_" --format="{{.ID}}")`

coredns versus systemd

Systemd likes to reinvent how Linux handles things. This does not exclude DNS resolvers:

...

pidof systemd && echo "yes" || echo "no"
ping $(hostname -f) && echo "yes" || echo "no"

Snap installs

snap install helm --classic
snap install kubectl --classic
export PATH=$PATH:/snap/bin

This is quick and dirty. In production: never use Snapd.

Start minikube

sudo minikube start --driver=none \
--extra-config=apiserver.enable-admission-plugins="LimitRanger,NamespaceExists,NamespaceLifecycle,ResourceQuota,ServiceAccount,DefaultStorageClass,MutatingAdmissionWebhook" \
--memory 9000 \
--cpus 4

sudo chown -R $USER ~/.kube
sudo chown -R $USER ~/.minikube

...

kubectl get nodes
kubectl get services -A --watch

watch -n 0.3 \
kubectl get po,services,deployment,rc,rs,ds,no,job,cm,ing -A # or similar

Basics: what is all this stuff? Recap on Kubernetes

The idea to placing the theory here is to allow a pragmatic and simplified reflection of the Kubernetes related concepts. At this point, it can be combined with a systematic exploration.

...

• If persistent storage is required, it’s usually better to use network file systems and to mount these external resources into the transient instances.

• Labels and Selectors are important if you have many Services. You may have different settings for different environments (test, prod, staging, … lab)

On-premise Serverless stack - Istio and Knative

Install Istio with the observability tools

Istio is like an extension to Kubernetes. It adds a Service Mesh. In my opinion, this is the future of how we are going to run and think about services in Cloud-Native environments.

...

Keep these variables (or the commands). We are going to need them to test the routes and reachability of our Egress endpoints of the Service Mesh (with curl).

Connect to Kiali with the browser from your Dev system without a tunnel

Kiali is a wonderful way to gain instant familiarity with a Micro Service Mesh Network. It uses Prometheus (a metrics collector) and Jaeger (a tracer).

With both systems combined, you can gain insights into the auto-scaling behavior of Knative services. This relates to Istio in so far, that its Sidecar containers get injected into the Knative service Pods, and that the metrics are provided this way.
– But let’s focus on one thing at a time. Before we are ready to observe all of this, we need to continue with the setup. Keep in mind: you don’t just want the front-row seats for this. You want to be a player!

Although minikube has tunneling and kubectl has got proxyfication features, it may be more comfortable to map Kiali to the VM’s IP. This could also be done with Ingress or MetalLB, but the simplest form of mapping out the service is this:

kubectl patch service kiali -n istio-system -p '{"spec": {"type": "LoadBalancer", "externalIPs":["'$(minikube ip)'"]}}'

Consider this specific step replaceable, but keep the one-liner in mind unless you prefer Ingress or MetalLB.

Add the gateway - only in a Dev env

Knative needs this because its routing (by default) will assume a real Kube cluster.

cd ~/istio-1.5.1 
helm template --namespace=istio-system \
  --set gateways.custom-gateway.autoscaleMin=1 \
  --set gateways.custom-gateway.autoscaleMax=2 \
  --set gateways.custom-gateway.cpu.targetAverageUtilization=60 \
  --set gateways.custom-gateway.labels.app='cluster-local-gateway' \
  --set gateways.custom-gateway.labels.istio='cluster-local-gateway' \
  --set gateways.custom-gateway.type='ClusterIP' \
  --set gateways.istio-ingressgateway.enabled=false \
  --set gateways.istio-egressgateway.enabled=false \
  --set gateways.istio-ilbgateway.enabled=false \
  --set global.mtls.auto=false \
  install/kubernetes/helm/istio \
  -f install/kubernetes/helm/istio/example-values/values-istio-gateways.yaml \
  | sed -e "s/custom-gateway/cluster-local-gateway/g" -e "s/customgateway/clusterlocalgateway/g" \
  > ./istio-local-gateway.yaml

kubectl apply -f istio-local-gateway.yaml

Patch the Istio Ingress-gateway

Again, this is just for the lab. Naturally, if you were to use OpenShift this would be a little different.

kubectl patch service istio-ingressgateway -n istio-system -p '{"spec": {"type": "LoadBalancer", "externalIPs":["'$(minikube ip)'"]}}'

kubectl apply -f https://github.com/knative/serving-operator/releases/download/v0.13.0/serving-operator.yaml 

cat <<-EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
 name: knative-serving
---
apiVersion: operator.knative.dev/v1alpha1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
EOF

kubectl get deployment knative-serving-operator

kubectl apply -f https://github.com/knative/eventing-operator/releases/download/v0.13.0/eventing-operator.yaml

cat <<-EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
 name: knative-eventing
---
apiVersion: operator.knative.dev/v1alpha1
kind: KnativeEventing
metadata:
  name: knative-eventing
  namespace: knative-eventing
EOF

kubectl get deployment -n knative-serving

Recap: what are Operators, ConfigMaps and Routes?

At this point (Q2 2020), there are different kinds of Operators that can extend the Kubernetes architecture. The topic is vast and involves Custom Resource Definitions (CRDs) that can extend the set of functions of the Kube API. Knative brings an operator, that will initialize an “Autoscaler”. Keep in mind that in production you may not want your API endpoints to scale down to 0 instances, and that you can configure all of that. For demonstration purposes, I refrained from doing this here. It’s super easy to do, via a ConfigMap object.

ConfigMaps simply hold the configuration similar to Linux’s /etc/ directory. They can be edited, and the respective Operators observe them. So in case you want to change Knative’s domain or the autoscaling behavior, you will find documented variables in the ConfigMap.

Routes are how Services are bridged out, either via custom LoadBalancers, Ingress, or other Service objects. Knative changes the Routes for the autoscaling to distribute the requests. This is visualized in a video in this essay.

The good news is, that you’ll pick this up in great detail over time. I find very little use for encyclopedic summaries about Kubernetes. This is an area of rapid progress and in a couple of months, everything here will be different again. If you are a Linux user, you will many opportunities to harness your command-line skills to pick things up on the fly.

Time for Hello World - Knative Go

Here we build, tag and push the local image. You can use Docker or Podman to do so.

cd ~/Source/knative-docs/docs/serving/samples/hello-world/helloworld-go
docker build .
docker login
docker tag e...6 wishi/knative-go-hello

docker push wishi/knative-go-hello
The push refers to repository [docker.io/wishi/knative-go-hello]
...

Time for 1, 2, 3, many curls

Let’s make a few HTTP requests to the service in parallel:

export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')
export INGRESS_HOST=$(minikube ip)

while true; \
do curl http://$INGRESS_HOST:$INGRESS_PORT  \
--header 'Host: helloworld-go.default.example.com' \
&& sleep 0.3; \
done
Hello Go Sample v1!
Hello Go Sample v1!
Hello Go Sample v1!

sudo apt-get install parallel
seq 1000 | parallel -n0 \
-j10 "curl -s http://INGRESS_HOST:$INGRESS_PORT \
-H 'Host: helloworldgo.default.example.com'"
Hello Go Sample v1!
Hello Go Sample v1!
Hello Go Sample v1!
...

Development of a Kotlin SpringBoot REST server into your Knative cluster

Let’s deliver Chuck Norris jokes at scale, via a REST service endpoint developed with Spring Boot, written in Kotlin.
The advantage of this is that the code is easy to read and that the service behavior will be compliant to the standards so that we can do HTTP load-testing. This way, we can see auto-scaling Serverless workloads in action.

I use a Maven package that provides the joke texts via Spring Boot Actuator package. This way I do not need a DBMS.

The application is rather primitive. It offers two endpoints:

• / - returns HTML, via a Thymeleaf template

• /joke - returns JSON, via a library and Spring

https://code.because-security.com/marius/chucknorris-kotlin-spring

I published the image to DockerHub. You can also build it yourself and push it into your own registry.

...

I would like to highlight that no code changes are required to run this application in Knative (or Docker). The code remains untouched.

Java 11 (versions higher than 8) is aware of Linux’s cgroups limits within container environments. This way, the container environment's memory limit is respected by the Java runtime.

– And you do not need to add mystical /dev/urandom mounts anymore. Just as a side-note.

...

marius@shell:~/Source/chucknorris-kotlin-spring$ kubectl apply -f service.yaml 
service.serving.knative.dev/chuckjokes-java-spring created

Now get the endpoint:

marius@shell:~/ curl -i http://$INGRESS_HOST:$INGRESS_PORT/joke \
-H "Accept: application/json" \
-H 'Host: chuckjokes-java-spring.default.example.com' \
-w "\n"
HTTP/1.1 200 OK
content-length: 55
content-type: application/json
date: Sun, 19 Apr 2020 06:56:19 GMT
server: istio-envoy
x-envoy-upstream-service-time: 7031

{"joke #1":"Chuck Norris's first program was kill -9."}

Initially, it will be slow because it’s instantiating the application on-demand. You can use this to warm up the cluster 🔥

I tried to get an Ubuntu VM to make more than 200 curl requests using GNU parallel. Although I unset the limits, I reached up to 162 requests per second, which is not enough to put the respective cluster under enough stress (the default requests per second to trigger the Knative autoscaler is 200:

marius@shell:~/$ kubectl -n knative-serving describe cm config-autoscaler

# The requests per second (RPS) target default is what the Autoscaler will
# try to maintain ...
requests-per-second-target-default: "200"

Instead, I used wrk, also because its invocation is similar to curl.

https://github.com/wg/wrk

marius@shell:~/$ wrk -t12 -c400 -d300s \
http://$INGRESS_HOST:$INGRESS_PORT/joke \
-H 'Host: chuckjokes-java-spring.default.example.com'

On the right, you can see the Pods serving Chuck Norris jokes. On the left, you can see the curl command to bring up the initial instance. This is followed by the wrk command to put the Service under enough load.

...