Kubernetes Response Engine, Part 2: Falcosidekick + OpenFaas
This blog post is part of a series of articles about how to create a
Kubernetes
response engine withFalco
,Falcosidekick
and aFaaS
.See other posts:
- Kubernetes Response Engine, Part 1 : Falcosidekick + Kubeless
- Kubernetes Response Engine, Part 3 : Falcosidekick + Knative
- Kubernetes Response Engine, Part 4 : Falcosidekick + Tekton
- Kubernetes Response Engine, Part 5 : Falcosidekick + Argo
- Kubernetes Response Engine, Part 6 : Falcosidekick + Cloud Run
- Kubernetes Response Engine, Part 7: Falcosidekick + Cloud Functions
We recently talked about a concept called "Kubernetes Response Engine", and we achieved this by using Falco
Falcosidekick
+Kubeless
. But as you might guess,Falcosidekick
project is evolving day after day, which means new outputs are added. With the release2.22.0
, we are proud to supportOpenFaaS
as a new output for Falcosidekick. This allows us to achieve the same concept, "Kubernetes Response Engine", but this time by using "OpenFaaS" instead of "Kubeless".
In this blog post, we will explain the basic concepts for integrating your own Response Engine into K8S with the
stack Falco
+ Falcosidekick
+ OpenFaaS
.
Prerequisites
We need tools with the following minimum versions to achieve this demo:
- Minikube v1.19.0
- Helm v3.5.3
- kubectl v1.21.0
- arkade v0.7.13
- faas-cli v0.13.9
Provision local Kubernetes Cluster
There are various ways to provision a local Kubernetes cluster such as, KinD, k3s, k0s, Minikube etc. We are going to use Minikube in this walkthrough.
Let's get provisioned a local Kubernetes cluster:
$ minikube start --cpus 3 --memory 8192 --vm-driver virtualbox
😄 minikube v1.19.0 on Darwin 10.15.7
✨ Using the virtualbox driver based on user configuration
👍 Starting control plane node minikube in cluster minikube
🔥 Creating virtualbox VM (CPUs=3, Memory=8192MB, Disk=20000MB) ...
🐳 Preparing Kubernetes v1.20.2 on Docker 20.10.4 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
Install OpenFaaS
OpenFaaS can be deployed into a variety of container orchestrators like Kubernetes, OpenShift, Docker Swarm or into a single host with faasd.
Follow the official documentation for deploying OpenFaaS to Kubernetes.
The fastest option is the tool called arkade to deploy OpenFaaS:
$ arkade install openfaas
Using Kubeconfig: /Users/batuhan.apaydin/.kube/config
Client: x86_64, Darwin
2021/04/10 21:39:29 User dir established as: /Users/batuhan.apaydin/.arkade/
"openfaas" already exists with the same configuration, skipping
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "falcosecurity" chart repository
...Successfully got an update from the "openfaas" chart repository
Update Complete. ⎈Happy Helming!⎈
VALUES values.yaml
Command: /Users/batuhan.apaydin/.arkade/bin/helm [upgrade --install openfaas openfaas/openfaas --namespace openfaas --values /var/folders/pf/6h9t0mnd4d342ncgpjq_3zl80000gp/T/charts/openfaas/values.yaml --set queueWorker.replicas=1 --set queueWorker.maxInflight=1 --set clusterRole=false --set operator.create=false --set faasnetes.imagePullPolicy=Always --set basicAuthPlugin.replicas=1 --set gateway.replicas=1 --set gateway.directFunctions=false --set openfaasImagePullPolicy=IfNotPresent --set ingressOperator.create=false --set basic_auth=true --set serviceType=NodePort]
Release "openfaas" does not exist. Installing it now.
NAME: openfaas
LAST DEPLOYED: Sat Apr 10 21:39:37 2021
NAMESPACE: openfaas
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
To verify that openfaas has started, run:
kubectl -n openfaas get deployments -l "release=openfaas, app=openfaas"
=======================================================================
= OpenFaaS has been installed. =
=======================================================================
# Get the faas-cli
curl -SLsf https://cli.openfaas.com | sudo sh
# Forward the gateway to your machine
kubectl rollout status -n openfaas deploy/gateway
kubectl port-forward -n openfaas svc/gateway 8080:8080 &
# If basic auth is enabled, you can now log into your gateway:
PASSWORD=$(kubectl get secret -n openfaas basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode; echo)
echo -n $PASSWORD | faas-cli login --username admin --password-stdin
faas-cli store deploy figlet
faas-cli list
# For Raspberry Pi
faas-cli store list \
--platform armhf
faas-cli store deploy figlet \
--platform armhf
# Find out more at:
# https://github.com/openfaas/faas
Thanks for using arkade!
Check if everything is working before moving onto the next step:
$ kubectl get pods --namespace openfaas
NAME READY STATUS RESTARTS AGE
alertmanager-74f9b48464-7gvrj 1/1 Running 0 2m13s
basic-auth-plugin-54bbd886f5-fclgn 1/1 Running 0 2m13s
gateway-6f8f5d5c87-tbxns 2/2 Running 0 2m13s
nats-695bf7587-hcbc2 1/1 Running 0 2m13s
prometheus-577c65f58c-4nvm7 1/1 Running 0 2m13s
queue-worker-b45b85966-g7kpt 1/1 Running 0 2m13s
Now, it is time to deploy our function. The function we are going to deploy basically receives events for an infected
pod from the Falcosidekick and deletes it immediately. Before deploying the function we need some
permissions to delete Pod. We create a ServiceAccount
with right to delete a Pod in any namespace, and we'll associate
it to our function:
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
name: falco-pod-delete
namespace: openfaas-fn
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: falco-pod-delete-cluster-role
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "delete"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: falco-pod-delete-cluster-role-binding
roleRef:
kind: ClusterRole
name: falco-pod-delete-cluster-role
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: falco-pod-delete
namespace: openfaas-fn
EOF
Now, we are ready to deploy our falco-pod-delete function, log in into OpenFaaS Gateway first:
$ kubectl port-forward -n openfaas svc/gateway 8080:8080 &
$ PASSWORD=$(kubectl get secret -n openfaas basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode; echo)
$echo -n $PASSWORD | faas-cli login --username admin --password-stdin
Calling the OpenFaaS server to validate the credentials...
credentials saved for admin http://127.0.0.1:8080
Install Falco + Falcosidekick
Firstly, we'll create the namespace that will host both Falco
and Falcosidekick
:
$ kubectl create namespace falco
We add the helm
repo:
$ helm repo add falcosecurity https://falcosecurity.github.io/charts
"falcosecurity" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "falcosecurity" chart repository
...Successfully got an update from the "openfaas" chart repository
Update Complete. ⎈Happy Helming!⎈
In a real project, you should get the whole chart with helm pull falcosecurity/falco --untar
and then configure
the values.yaml
. For this tutorial, will try to keep thing as easy as possible and set configs directly
by passing arguments to helm install
command line:
$ helm upgrade --install falco falcosecurity/falco --namespace falco \
--set falcosidekick.enabled=true \
--set falcosidekick.config.openfaas.functionname="falco-pod-delete"
You should get this output:
Release "falco" does not exist. Installing it now.
NAME: falco
LAST DEPLOYED: Tue Apr 13 10:49:49 2021
NAMESPACE: falco
STATUS: deployed
REVISION: 1
NOTES:
Falco agents are spinning up on each node in your cluster. After a few
seconds, they are going to start monitoring your containers looking for
security issues.
No further action should be required.
And you can see your new Falco
and Falcosidekick
pods:
$ kubectl get pods --namespace falco
NAME READY STATUS RESTARTS AGE
falco-falcosidekick-7779579477-mwsb4 1/1 Running 0 67s
falco-falcosidekick-7779579477-n5v89 1/1 Running 0 67s
falco-p97rw 1/1 Running 0 67s
The argument falcosidekick.enabled=true
sets the following settings in Falco for you:
--set falco.jsonOutput=true \
--set falco.httpOutput.enabled=true \
--set falco.httpOutput.url=http://falco-falcosidekick:2801
The
arguments --set falco.jsonOutput=true --set falco.httpOutput.enabled=true --set falco.httpOutput.url=http://falco-falcosidekick:2801
are there to configure the format of events and the URL where Falco
will send them. As Falco
and Falcosidekick
will
be in the same namespace, it can directly use the name of the service (falco-falcosidekick
) above Falcosidekick
pods.
We check the logs:
$ kubectl logs deployment/falco-falcosidekick -n falco
Found 2 pods, using pod/falcosidekick-5c696d7fd8-9bnnj
2021/04/10 19:21:55 [INFO] : Enabled Outputs : [OpenFaaS]
2021/04/10 19:21:55 [INFO] : Falco Sidekick is up and listening on :2801
OpenFaaS
is displayed as enabled output, everything is good 👍.
Install our OpenFaaS function
Our really basic function will receive events from Falco
thanks to Falcosidekick
, check if the triggered rule is *
Terminal Shell in container*
, extract the namespace and pod name from the fields of events and delete the according pod:
Basically, the process is:
+----------+ +---------------+ +----------+
| Falco +-----------------> Falcosidekick +--------------------> OpenFaaS |
+----^-----+ sends event +---------------+ triggers +-----+----+
| |
detects a shell | |
| |
+----+-------+ deletes |
| Pwned Pod <----------------------------------------------------------+
+------------+
Let's create the function and deploy it:
$ faas-cli template store pull golang-middleware
Fetch templates from repository: https://github.com/openfaas/golang-http-template at
2021/04/10 21:56:34 Attempting to expand templates from https://github.com/openfaas/golang-http-template
2021/04/10 21:56:35 Fetched 2 template(s) : [golang-http golang-middleware] from https://github.com/openfaas/golang-http-template
$ tree -L 2 .
.
└── template
├── golang-http
└── golang-middleware
# Don't forget to set your docker id in the prefix section, mine is devopps.
$ faas-cli new falco-pod-delete --lang golang-middleware --prefix devopps
faas-cli new falco-pod-delete --lang golang-middleware --prefix devopps
Folder: falco-pod-delete created.
___ _____ ____
/ _ \ _ __ ___ _ __ | ___|_ _ __ _/ ___|
| | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \
| |_| | |_) | __/ | | | _| (_| | (_| |___) |
\___/| .__/ \___|_| |_|_| \__,_|\__,_|____/
|_|
Function created in folder: falco-pod-delete
Stack file written: falco-pod-delete.yml
Notes:
You have created a new function which uses Golang 1.13.
To include third-party dependencies, use Go modules and use
"--build-arg GO111MODULE=on" with faas-cli build or configure this
via your stack.yml file.
See more: https://docs.openfaas.com/cli/templates/
For detailed examples:
https://github.com/openfaas-incubator/golang-http-template
$ tree -L 2 .
.
├── falco-pod-delete
│ └── handler.go
├── falco-pod-delete.yml
└── template
├── golang-http
└── golang-middleware
First, replace the falco-pod-delete.yml with the following content:
version: 1.0
provider:
name: openfaas
gateway: http://127.0.0.1:8080
functions:
falco-pod-delete:
lang: golang-middleware
handler: ./falco-pod-delete
image: devopps/falco-pod-delete:latest # be careful this line, it should be your docker id.
annotations:
com.openfaas.serviceaccount: falco-pod-delete
build_args:
GO111MODULE: on
Once you have edited it, let's continue with the code, create a go.mod
.
$ cd falco-pod-delete
$ go mod init falco-pod-delete
go: creating new go.mod: module falco-pod-delete
go: to add module requirements and sums:
go mod tidy
Then, replace the handler.go
with the following content:
package function
import (
"context"
"encoding/json"
"io/ioutil"
"log"
"net/http"
"time"
metaV1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/rest"
)
var kubeClient *kubernetes.Clientset
func init() {
// creates the in-cluster config
config, err := rest.InClusterConfig()
if err != nil {
panic(err.Error())
}
// creates the clientset
kubeClient, err = kubernetes.NewForConfig(config)
if err != nil {
panic(err.Error())
}
}
type Alert struct {
Output string `json:"output"`
Priority string `json:"priority"`
Rule string `json:"rule"`
Time time.Time `json:"time"`
OutputFields struct {
ContainerID string `json:"container.id"`
ContainerImageRepository interface{} `json:"container.image.repository"`
ContainerImageTag interface{} `json:"container.image.tag"`
EvtTime int64 `json:"evt.time"`
FdName string `json:"fd.name"`
K8SNsName string `json:"k8s.ns.name"`
K8SPodName string `json:"k8s.pod.name"`
ProcCmdline string `json:"proc.cmdline"`
} `json:"output_fields"`
}
var CriticalNamespaces = []string{"kube-system", "kube-public", "kube-node-lease", "falco", "openfaas", "openfaas-fn"}
func Handle(w http.ResponseWriter, r *http.Request) {
var alert Alert
if r.Body != nil {
defer r.Body.Close()
body, _ := ioutil.ReadAll(r.Body)
json.Unmarshal(body, &alert)
podName := alert.OutputFields.K8SPodName
namespace := alert.OutputFields.K8SNsName
var critical bool
for _ , ns := range CriticalNamespaces {
if ns == namespace {
critical = true
break
}
}
if !critical {
log.Printf("Deleting pod %s from namespace %s", podName, namespace)
kubeClient.CoreV1().Pods(namespace).Delete(context.Background(), podName, metaV1.DeleteOptions{})
}
}
w.WriteHeader(http.StatusOK)
w.Write([]byte("OK"))
}
After that, update your Go Modules by doing go mod tidy
:
$ go mod tidy
go: finding module for package k8s.io/client-go/rest
go: finding module for package k8s.io/apimachinery/pkg/apis/meta/v1
go: finding module for package k8s.io/client-go/kubernetes
go: downloading k8s.io/client-go v0.21.0
go: downloading k8s.io/apimachinery v0.21.0
go: found k8s.io/apimachinery/pkg/apis/meta/v1 in k8s.io/apimachinery v0.21.0
go: found k8s.io/client-go/kubernetes in k8s.io/client-go v0.21.0
go: found k8s.io/client-go/rest in k8s.io/client-go v0.21.0
go: downloading k8s.io/api v0.21.0
go: downloading golang.org/x/net v0.0.0-20210224082022-3d97a244fca7
go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.1.0
go: downloading golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d
Now, you should be able to build, push and deploy your function with faas-cli
:
$ cd ..
$ faas-cli up -f falco-pod-delete.yml
[0] > Building falco-pod-delete.
Clearing temporary build folder: ./build/falco-pod-delete/
Preparing: ./falco-pod-delete/ build/falco-pod-delete/function
Building: devopps/falco-pod-delete:latest with golang-middleware template. Please wait..
#1 [internal] load build definition from Dockerfile
#1 sha256:8cd765381aabb90df3bcfbc06f4d175af37d66b85125d463585abc1fc878b94b
#1 transferring dockerfile: 1.81kB done
#1 DONE 0.0s
...
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Image: devopps/falco-pod-delete:latest built.
[0] < Building falco-pod-delete done in 1.31s.
[0] Worker done.
Total build time: 1.31s
[0] > Pushing falco-pod-delete [devopps/falco-pod-delete:latest].
The push refers to repository [docker.io/devopps/falco-pod-delete]
8096edd09fbc: Layer already exists
464d68aca3d9: Layer already exists
e4766ea46ad0: Layer already exists
5f70bf18a086: Layer already exists
a823d50a5b72: Layer already exists
060f21486264: Layer already exists
8ea3b23f387b: Layer already exists
latest: digest: sha256:f94abba203232b97cb2873ef5d60eec31b52d492f3d3ee106d6a9877bf131d95 size: 1782
[0] < Pushing falco-pod-delete [devopps/falco-pod-delete:latest] done.
[0] Worker done.
Deploying: falco-pod-delete.
Deployed. 202 Accepted.
URL: http://127.0.0.1:8080/function/falco-pod-delete
Check if everything is working before moving to the next step:
$ kubectl get pods --namespace openfaas-fn
NAME READY STATUS RESTARTS AGE
falco-pod-delete-7dc9f5fbb8-gbfk7 1/1 Running 0 27s
Test our function
We start by creating a dumb pod:
$ kubectl run alpine --namespace default --image=alpine --restart='Never' -- sh -c "sleep 600"
pod/alpine created
$ kubectl get pods --namespace default
AME READY STATUS RESTARTS AGE
alpine 1/1 Running 0 11s
Let's run a shell command inside and see what happens:
$ kubectl exec -i --tty alpine --namespace default -- sh -c "uptime"
19:27:21 up 50 min, load average: 0.11, 0.12, 0.11
As expected we got the result of our command, but, if we get the status of the pod we retrieve:
$ kubectl get pods --namespace default
NAME READY STATUS RESTARTS AGE
alpine 1/1 Terminating 0 103s
💥 It has been terminated 💥
We can now check the logs of components.
For Falco
:
$ kubectl logs daemonset/falco --namespace falco
..
{"output":"19:27:21.002873265: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=default k8s.pod=alpine container=97c9868ea832 shell=sh parent=runc cmdline=sh -c uptime terminal=34816 container_id=97c9868ea832 image=alpine) k8s.ns=default k8s.pod=alpine container=97c9868ea832","priority":"Notice","rule":"Terminal shell in container","time":"2021-04-10T19:27:21.002873265Z", "output_fields": {"container.id":"97c9868ea832","container.image.repository":"alpine","evt.time":1618082841002873265,"k8s.ns.name":"default","k8s.pod.name":"alpine","proc.cmdline":"sh -c uptime","proc.name":"sh","proc.pname":"runc","proc.tty":34816,"user.loginuid":-1,"user.name":"root"}}
{"output":"19:27:21.038853452: Notice Unexpected connection to K8s API Server from container (command=handler k8s.ns=openfaas-fn k8s.pod=falco-pod-delete-7dc9f5fbb8-gbfk7 container=12fc4de5ccc3 image=devopps/falco-pod-delete:latest connection=172.17.0.9:43812->10.96.0.1:443) k8s.ns=openfaas-fn k8s.pod=falco-pod-delete-7dc9f5fbb8-gbfk7 container=12fc4de5ccc3","priority":"Notice","rule":"Contact K8S API Server From Container","time":"2021-04-10T19:27:21.038853452Z", "output_fields": {"container.id":"12fc4de5ccc3","container.image.repository":"devopps/falco-pod-delete","container.image.tag":"latest","evt.time":1618082841038853452,"fd.name":"172.17.0.9:43812->10.96.0.1:443","k8s.ns.name":"openfaas-fn","k8s.pod.name":"falco-pod-delete-7dc9f5fbb8-gbfk7","proc.cmdline":"handler"}}
For Falcosidekick
:
$ kubectl logs deployment/falcosidekick --namespace falco
2021/04/10 19:27:21 [INFO] : OpenFaas - Post OK (200)
2021/04/10 19:27:21 [INFO] : OpenFaas - Function Response : OK
2021/04/10 19:27:21 [INFO] : OpenFaas - Call Function "falco-pod-delete.openfaas-fn" OK
..
For falco-delete-pod function:
$ faas-cli logs -f --name falco-pod-delete
2021/04/10 19:34:03 Deleting pod alpine from namespace default
2021/04/10 19:34:03 POST / - 200 OK - ContentLength: 2
Conclusion
With this really simple example, we only scratched the surface of possibilities, so don't hesitate to share with us on Slack (https://kubernetes.slack.com #falco) your comments, ideas and successes. You're also always welcome to contribute.