Prometheus, Grafana and Minikube

Context:

The idea was to install Prometheus and Grafana at YTI so we can troubleshoot stuff. Several deployment options were considered and the one with K8s was selected. The cluster creation and installation was based on this one: https://www.chevdor.com/post/2021/02/docker_to_k8s/ because it touched several points that others did not:

remote control of the k8s via api
uses minikube as k8s implementation

Prerequisites:

kubectl installed on local machine (windows, linux, mac)
some degree of familiarity with kubernetes and docker and linux

Steps

As usual not every step described in the tutorial worked smoothly, but going through it step by step and having check points along the way made it work. Where in doubt and when something does not work, best is to refer to the official web page of the component that’s not working

Below are the stages of the installation:

Get a server: a Centos7 VM was provided. Thanks @User
Install packages on the server: on top of the installation provided, the following would need to be installed:
kubectl - this allows you to interact with the k8s cluster locally although is not necessary https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-using-native-package-management
minikube - this is the local k8s: https://minikube.sigs.k8s.io/docs/start/

Latest minikube version seems to have some issues with the core DNS. To get it to work one may need a specific minikube version

curl -LO https://github.com/kubernetes/minikube/releases/download/v1.25.2/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube
$ minikube version
minikube version: v1.25.2

docker - the container manager used by minikube: https://docs.docker.com/engine/install/centos/

Latest minikube does not work with earlier versions of docker. In the end I had to install

Docker version 20.10.23

When attempting to run minikube or docker as the current user an error message will be raised (together with the solution which is to basically add the current user to the docker group):

sudo usermod -aG docker $USER && newgrp docker

Prepare the server
- start the cluster with the external ips specified (this will generate the certs used to connect remotely) and with a service cluster ip range different than default (the default value may conflict with the actual subnet used by the host)
  minikube start --apiserver-ips=10.100.60.70 --service-cluster-ip-range=172.16.0.0/16
- install a reverse proxy that bridges the outside to the internal cluster network:

docker run -d --network minikube -p 18443:18443 chevdor/nginx-minikube-proxy

Export some local environment variables and copy the authentication data from remote to local machine

export MY_MINIKUBE=~/k8s/minikube/test
export MY_IP=10.100.60.70
export MY_SRV=ytpindo1javamon1
scp minikube:~/.minikube/ca.crt $MY_MINIKUBE
scp minikube:~/.minikube/profiles/minikube/client.{key,crt} $MY_MINIKUBE

Configure connection to another kubernetes cluster (remote minikube) on your local machine

The steps below will affect the connection to your local minikube. That's because it reuses the minikube user. Something that needs to be looked into if it presents a problem or if having both minikube installations is a need.

Add a cluster to your local kubectl config

kubectl config set-cluster $MY_SRV --server https://$MY_IP:18443 --certificate-authority=$MY_MINIKUBE/ca.crt

Out cluster is now known:

kubectl config get-clusters | grep $MY_SRV

Reconfigure the credentials for minikube user:

kubectl config set-credentials minikube --client-certificate=$MY_MINIKUBE/client.crt --client-key=$MY_MINIKUBE/client.key

Configure all as a context context:

kubectl config set-context $MY_SRV --cluster=$MY_SRV --namespace=default --user=minikube

Use the newly created context:

kubectl config use-context $MY_SRV