As we slowly move away from Docker Swarm, we have set our sights on implementing a Kubernetes cluster that we can use to build, test and deploy our applications on. This guide explains how to create a Kubernetes cluster from scratch using CentOS 7 virtual machines.
Prerequisites
To follow this guide, you require:
- Three CentOS 7 servers (virtual or physical) connected to the same network. One server will be the Kubernetes Master/Manager node, and the other two servers will be Kubernetes worker nodes.
- A Jenkins instance with the Kubernetes-Plugin installed.
- A basic understanding of Containerisation.
- A basic understanding of Kubernetes: Containers, Pods and Services.
Kubernetes: setting up the master node
Follow the steps below:
1. Disable SELinux:
setenforce 0 sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux
2. Disable swapping on all devices:
swapoff -a
3. Configure CentOS’s firewall (FirewallD):
firewall-cmd --permanent --add-port=6443/tcp firewall-cmd --permanent --add-port=2379-2380/tcp firewall-cmd --permanent --add-port=10250/tcp firewall-cmd --permanent --add-port=10251/tcp firewall-cmd --permanent --add-port=10252/tcp firewall-cmd --permanent --add-port=10255/tcp firewall-cmd --reload
4. Configure iptables and networking protocols:
modprobe br_netfilter echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables
3. Add the Kubernetes Google Repository as a Yum repository:
cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF
4. Install Kubeadm and Docker, and start them as a services. Optionally, you can also update all packages installed on your machine:
cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF
5. If you know that you will need to access an insecure Docker registry, add them to the Docker daemon and restart the service:
cat <<EOF > /etc/docker/daemon.json
{
"insecure-registries" : [
"docker.someplace.caplin.com"
]
}
EOF
systemctl restart docker
6. Start Kubeadm, and create the relevant directories it instructs you to make. In Kubeadm’s output is a link that you will need to connect the worker nodes to your master within the cluster; make a note of the link.
kubeadm init mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
7. Deploy Weave Net as the Pod Network Provider:
export kubeVersion=$(kubectl version | base64 | tr -d '\n') kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$kubeVersion"
8. Check that Weave Net and other Kube-System pods are runnning correctly by entering the following:
kubectl get pods --all-namespaces
You should see a list of all the pods running in your cluster
9. Check that the Kubernetes Master and KubeDNS are working correctly by entering the following:
kubectl cluster-info kubectl get nodes
You should see:
- The IP which the Kubernetes master is running at
- The address of the KubeDNS
- A table listing the nodes in your cluster with the Status, Role, Age and Version
10. Create a default service account. This is used by the Jenkins Kubernetes-Plugin to manage our API calls from the pods we spawn:
kubectl create clusterrolebinding --user system:serviceaccount:default:default kube-system-cluster-admin --clusterrole cluster-admin
Kubernetes: setting up the worker nodes
For each of the remaining Centos 7 virtual machines, do the following:
1. Follow steps 1-5 mentioned in the Master/Manager setup.
2. Enter the join command that the Master/Manager node gave you when you started kubeadm. It should have the format:
kubeadm join --token ... --discovery-token-ca-cert-hash ...
3. SSH back onto your Master/Manager node and see whether the worker node has connected by entering the following:
kubectl get nodes
Jenkins: configuring the Kubernetes Plugin
Follow the steps below:
1. On your Manager/Master node, navigate to your home directory and then enter the .kube directory
2. Within this directory there should be a config file. Within that file there is a config file. Look for the key that is called “certificate-authority-data”
3. Take the value of the key and insert it into the following command:
echo <certificate-authority-data-key> | base64 -d > ca.crt
This will create a key in a file called ca.crt. Make note of this new key; you’ll need it later.
4. Run the following command on the Master/manager:
kubectl -n default describe secret $(kubectl -n kube-system get secret | grep kubernetes-admin | awk '{print $1}')
This will return a token for the default service account we created earlier. Make note of this token as well.
5. Head to your Jenkins instance. Once there, click Credentials > System > Global Credentials > Add credentials .Create a credential based on a secret text. Add the token we got from from the default service as your secret (Step 4). Given it a suitable ID and description, then click OK to save.
6. Head back to the Jenkins home page and click Manage Jenkins > Configure System. Scroll all the way down to “Add new Cloud”, and click on Kubernetes. Here we will configure the plugin.
7. Enter the following details to configure the plugin:
- Name: ‘kubernetes’, unless you want to specifically define the cloud in our jenkins file.
- URL: the IP and port of your Kubernetes master/manager. You can find this by typing the following while on the master/manager:
kubectl cluster-info
- Kubernetes server certificate key: the key we generated in the ca.crt file earlier.
- Disable HTTPS certificate checks, if required.
- The Kubernetes namespace: leave this as ‘default’.
- Credentials: select the Kuberntees credentials we added earlier.
8. Click Test Connection. You should have confirmation that the connection test was successful
9. We’ll now setup the Jenkins information section of the plugin:
- Jenkins URL: the IP to your Jenkins instance
- Jenkins tunnel: the port the agents will need to connect to. You can find this by going to the Configure Global Security section and seeing the value of “TCP port for JNLP agents”. If this is disabled, put this as fixed with a value.
- Accept the remaining default field values. These can be modified later on once the you are up and running
10. Now we need to setup the default Kubernetes pod template for each time the plugin creates a pod in Kubernetes. This is used for creating a Jenkins agent that will connect to our Jenkins instance. This can be added by clicking “Add Pod Template” and by adding the following following configuration:
- Name = jnlp
- Namespace = default
- Usage = Use this node as much as possible
11. Next, we need to create a container template for the Jenkins Agents. Click “Add container” and enter the following values:
- Name = jnlp
- Docker image = jenkins/jnlp-slave:2.6.2
- Always Pull Image
- Working directory = /home/jenkins
- Command to run = /bin/sh -c
- Arguments to pass to the command = ${computer.jnlpmac}${computer.name}
- Allocate pseduo-TTY
12. Finally, we need to map the ports of our container to our Jenkins instance. Click Add port mapping and then make sure you have two “Container Exposed Ports”.
For the first one, add:
- Name = Agent
- ContainerPort = 50000
- HostPort = 15432
For the second one, add:
- Name = http
- ContainerPort = 8080
- HostPort = 8080
Jenkins: creating a Jenkinsfile
The Kubernetes plugin github page has several Jenkinsfile examples. Here is a slightly modified example I have annotated:
#!groovy
//Set the label of the pod we spin up as a unique ID to prevent pod interference
def ourPod = "${UUID.randomUUID().toString()}"
//Pull the maven image from the Docker repository.
//Ensure it remains running by executing 'cat' with no arguments.
podTemplate(label: ourPod, containers: [
containerTemplate(name: 'maven',
image: 'maven:3.3.9-jdk-8-alpine',
ttyEnabled: true,
command: 'cat'
)
]) {
//Run the build in our pod
node(ourPod) {
stage('Build a Maven project') {
git 'https://github.com/jenkinsci/kubernetes-plugin.git'
//Run this command within our maven container.
container('maven') {
sh 'mvn -B clean package'
}
}
}
}
By creating a pipeline job and running this, we can check whether the Jenkins agent connects to our Jenkins and the job executes what it should do within the Maven container.
Resources
- https://www.linuxtechi.com/install-kubernetes-1-7-centos7-rhel7/
- https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
- https://github.com/jenkinsci/kubernetes-plugin
- https://hub.docker.com/r/jenkins/slave/
- https://jenkins.io/doc/book/pipeline/