I want to install a OpenShift Cluster on my HP Proliant D160 G6 server at home. For me the biggest challenge was to organize all the required network settings.
Related to 2 great posts I get the cluster running and want to write a description as reminder for me .
https://itnext.io/install-openshift-4-2-on-kvm-1117162333d0
https://medium.com/@zhimin.wen/preparing-openshift-4-1-environment-dc40ecb7a763
I'm installing a OpenShift 4.3 nightly builds Cluster with 3 Master-Nodes and 2 Worker-Nodes. The VMs are build with KVM on Ubuntu 18.04
This description also works with OpenShift 4.2. I tested it many times ;-)
Following steps are neccessary:
- KVM Setup
- Dnsmasq set up and common configuration
- Configure Ubuntu to use local dnsmasq
- dnsmasq cluster specific configuration
- Loadbalancer and Gobetween configuration
- Webserver for installation files
- Create VMs with CoreOs Image
- Installation of OpenShift
- Traefik configuration to route the APi
KVM-Setup
Install required packages
$sudo apt -y install qemu qemu-kvm libvirt-bin bridge-utils virt-manager libosinfo-bin
Make sure the libvirtd service is enabled and running
Install the uvtool to create Ubuntu-VMs in an easy way
$sudo apt-get -y uvtool
$sudo uvt-simplestreams-libvirt --verbose sync release=bionic arch=amd64
Adding: com.ubuntu.cloud:server:18.04:amd64 20200107
$ssh-keygen -b 4096 -t rsa -f ~/.ssh/id_rsa -N ""
Create a KVM-network disabling DNS and DHCP in this network
net-ocp.xml
<network>
<name>ocp</name>
<forward mode='nat'/>
<bridge name='br-ocp' stp='on' delay='0'/>
<dns enable="no"/>
<ip address='192.168.10.1' netmask='255.255.255.0'>
</ip>
</network>
$virsh net-define net-ocp.xml
Network ocp defined from net-ocp.xml
$virsh net-autostart ocp
Network ocp marked as autostarted
$virsh net-start ocp
Network ocp started
$sudo systemctl restart libvirt-bin
The bridge ist created and configured:
$ifconfig br-ocp
br-ocp: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.10.1 netmask 255.255.255.0 broadcast 192.168.10.255
ether 52:54:00:dc:f2:c6 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Dnsmasq setup and common configurations
If you haven't already install dnsmasq do that and enable the service
$sudo apt-get install -y dnsmasq
Create a file /etc/dnsmaq.d/common.conf
# Listen on lo and br-ocp only
bind-interfaces
interface=lo,br-ocp
# DHCP
dhcp-option=option:router,192.168.10.1
dhcp-option=option:dns-server,192.168.10.1
dhcp-range=192.168.10.10,192.168.10.254,12h
# forward, use original DNS server
server=10.0.xxx.xxx
server=10.0.xxx.xxx
address=/webserver.oc.io/192.168.10.20
Configure Ubuntu to use local dnsmasq
Rename the /etc/resolve.conf (some application first ping this file)
$ sudo mv /etc/resolv.conf /etc/resolv.conf.bac
Backup /etc/systemd/resolved.conf and change the DNS entry to localhost
$sudo cp /etc/systemd/resolved.conf /etc/systemd/resolved.conf.bac
/etc/systemd/resolved.conf (new):
[Resolve]
DNS=127.0.0.1
$sudo systemctl restart systemd-resolved.service
$sudo systemd-resolve --status
Global
DNS Servers: 127.0.0.1
DNSSEC NTA: 10.in-addr.arpa
...
The KVM host now use the local dnsmasq as DNS server
DNS configuration for OpenShift Installation
Create a file /etc/dnsmasq.d/test-ocp4.conf
Inside this file define for each node of the cluster following entries:
- DHCP IP address by MAC address
- DNS A record
- DNS PTR record
My test-ocp4.conf looks like here:
# Configuration for OCP Cluster *.test-ocp4.oc.io
# Bootstrap
dhcp-host=52:54:00:76:34:04,192.168.10.40
address=/bootstrap.test-ocp4.oc.io/192.168.10.40
ptr-record=40.10.168.192.in-addr.arpa,bootstrap.test-ocp4.oc.io
# Master1
dhcp-host=52:54:00:28:4e:09,192.168.10.41
address=/master1.test-ocp4.oc.io/192.168.10.41
ptr-record=41.10.168.192.in-addr.arpa,master1.test-ocp4.oc.io
# Master2
dhcp-host=52:54:00:07:45:14,192.168.10.42
address=/master2.test-ocp4.oc.io/192.168.10.42
ptr-record=42.10.168.192.in-addr.arpa,master2.test-ocp4.oc.io
# Master3
dhcp-host=52:54:00:78:05:d9,192.168.10.43
address=/master3.test-ocp4.oc.io/192.168.10.43
ptr-record=43.10.168.192.in-addr.arpa,master3.test-ocp4.oc.io
# Worker1
dhcp-host=52:54:00:e4:67:14,192.168.10.51
address=/worker1.test-ocp4.oc.io/192.168.10.51
ptr-record=51.10.168.192.in-addr.arpa,worker1.test-ocp4.oc.io
# Worker2
dhcp-host=52:54:00:e5:c5:38,192.168.10.52
address=/worker2.test-ocp4.oc.io/192.168.10.52
ptr-record=52.10.168.192.in-addr.arpa,worker2.test-ocp4.oc.io
# LoadBalancer
dhcp-host=52:54:00:41:e5:45,192.168.10.49
address=/lb.test-ocp4.oc.io/192.168.10.49
ptr-record=49.10.168.192.in-addr.arpa,lb.test-ocp4.co.io
# LB settings for the API
address=/api.test-ocp4.oc.io/192.168.10.49
address=/api-int.test-ocp4.oc.io/192.168.10.49
address=/.apps.test-ocp4.oc.io/192.168.10.49
# ETCD instance A and SRV records
address=/etcd-0.test-ocp4.oc.io./192.168.10.41
srv-host=_etcd-server-ssl._tcp.test-ocp4.oc.io,etcd-0.test-ocp4.oc.io,2380
address=/etcd-1.test-ocp4.oc.io/192.168.10.42
srv-host=_etcd-server-ssl._tcp.test-ocp4.oc.io,etcd-1.test-ocp4.oc.io,2380
address=/etcd-2.test-ocp4.oc.io/192.168.10.43
srv-host=_etcd-server-ssl._tcp.test-ocp4.oc.io,etcd-2.test-ocp4.oc.io,2380
Before restart of the dnsmasq we may need to clear the lease chache, to make sure the DHCP IP allocation is not cached
$sudo rm -rf /var/lib/misc/dnsmasq.leases
$sudo touch /var/lib/misc/dnsmasq.leases
$sudo systemctl restart dnsmasq.service
Loadbalancer and Gobetween configuration
For Loadbalancer we install Gobetween in a minimal Ubuntu VM. So first create the VM
$uvt-kvm create lb release=bionic --memory 4096 --cpu 4 --disk 50 --bridge br-ocp --password password
$virsh list --all
Id Name State
----------------------------------------------------
1 lb running
$virsh dumpxml lb | grep 'mac address' | cut -d\' -f 2
52:54:00:8e:1d:32
Change the MAC address in the /etc/dnsmasq.d/test-ocp4.conf file.
Clear the lease cache and restart dnsmasq.
Restart the VM to make sure it gets the assigned IP
$virsh destroy lb
Domain lb destroyed
$virsh start lb
Domain lb started
$mkdir gobetween
$cd gobetween
$curl -LO https://github.com/yyyar/gobetween/releases/download/0.7.0/gobetween_0.7.0_linux_amd64.tar.gz
$tar xzvf gobetween_0.7.0_linux_amd64.tar.gz
AUTHORS
CHANGELOG.md
LICENSE
README.md
config/
config/gobetween.toml
gobetween
$sudo cp gobetween /usr/local/bin/
Create a systemd service (copy file beyond to /etc/systemd/system/gobetween.service)
gobetween.service
[Unit]
Description=Gobetween - modern LB for cloud era
Documentation=https://github.com/yyyar/gobetween/wiki
After=network.target
[Service]
Type=simple
PIDFile=/run/gobetween.pid
#ExecStartPre=prestart some command
ExecStart=/usr/local/bin/gobetween -c /etc/gobetween/gobetween.toml
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true
[Install]
WantedBy=multi-user.target
Create a configuration file for gobetween
/etc/gobetween/gobetween.toml
[servers]
[servers.api]
protocol = "tcp"
bind = "0.0.0.0:6443"
[servers.api.discovery]
kind = "static"
static_list = [
"192.168.10.40:6443","192.168.10.41:6443","192.168.10.42:6443","192.168.10.43:6443" ]
[servers.api.healthcheck]
kind = "ping"
fails = 1
passes = 1
interval = "2s"
timeout="1s"
ping_timeout_duration = "500ms"
[servers.mcs]
protocol = "tcp"
bind = "0.0.0.0:22623"
[servers.mcs.discovery]
kind = "static"
static_list = [ "192.168.10.40:22623","192.168.10.41:22623","192.168.10.42:22623","192.168.10.43:22623" ]
[servers.mcs.healthcheck]
kind = "ping"
fails = 1
passes = 1
interval = "2s"
timeout="1s"
ping_timeout_duration = "500ms"
[servers.http]
protocol = "tcp"
bind = "0.0.0.0:80"
[servers.http.discovery]
kind = "static"
static_list = [ "192.168.10.44:80","192.168.10.45:80","192.168.10.46:80" ]
[servers.http.healthcheck]
kind = "ping"
fails = 1
passes = 1
interval = "2s"
timeout="1s"
ping_timeout_duration = "500ms"
[servers.https]
protocol = "tcp"
bind = "0.0.0.0:443"
[servers.https.discovery]
kind = "static"
static_list = [ "192.168.10.44:443","192.168.10.45:443","192.168.10.46:443" ]
[servers.https.healthcheck]
kind = "ping"
fails = 1
passes = 1
interval = "2s"
timeout="1s"
ping_timeout_duration = "500ms"
Start the gobetween service
Webserver for installation files
Back to my server I have to install a webserver which will host the files which are necessary for the installation of the CoreOS VMs
$sudo apt-get install -y apache2
Check if the webserver is online
$ curl http://playground:80
There should be a output ;-) (playground is the name of my server)
Create VMs with CoreOs Image
To create CoreOS VMs we need first the ISO image and additional Ignition Files and an initramfs Image. Everything can be get from the Red Hat page with a Red Hat Account
Here you get the installer for the stable release and also the intaller for the nightly builds.
Download the installer
$wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/latest-4.3/openshift-install-linux-4.3.0-0.nightly-2020-01-25-074537.tar.gz
Download your pull secret
Download the RHCOS image
$wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/latest/rhcos-43.81.202001142154.0-installer.x86_64.iso
Download compressed metal BIOS
$wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/latest/rhcos-43.81.202001142154.0-metal.x86_64.raw.gz
Download the oc-client
$wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/latest-4.3/openshift-install-linux-4.3.0-0.nightly-2020-01-25-074537.tar.gz
To get the Ignition Files we have to take some manual steps:
1. unpack the installer in a new directory "install"
$tar xzvf openshift-install-linux-4.3.0-0.nightly-2020-01-25-074537.tar.gz
2. Create a install-config.yaml file (see also the documentation)
apiVersion: v1
baseDomain: oc.io
compute:
- hyperthreading: Enabled
name: worker
replicas: 0
controlPlane:
hyperthreading: Enabled
name: master
replicas: 3
metadata:
name: test-ocp4
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
networkType: OpenShiftSDN
serviceNetwork:
- 172.30.0.0/16
platform:
none: {}
pullSecret: '<pull-secret>'
sshKey: '<ssh-key>'
3. Create the Kubernetes manifest
$./openshift-install create manifests --dir=.
After this command, there are a lot of manifests created in the new directory "manifests"
4. Modify the manifest/cluster-scheduler-02-config.yml to prevent Pods from being scheduled on the control plane nodes. For this change the entry "masterSchedulable" from true to false
5. Create the Ignistion files
$./openshift-install create ignition-configs --dir=.
Now the Ignition Files for the bootstrap-, master-, and worker-nodes are present. Also the kubeconfig file and the kubeadmin password file is created under auth directory
6. Upload the Ignition-Files to the webserver
$ sudo cp bootstrap.ign master.ign worker.ign /var/www/html/
$ sudo chmod +r ./*
7. Upload the BIOS image to the webserver
$ sudo cp ../rhcos-43.81.202001142154.0-metal.x86_64.raw.gz /var/www/html/rhcos-43-metal.raw.gz
For creation of the VM I copied the ISO to the directory /var/lib/libvirt/images/
$ sudo mv ../rhcos-43.81.202001142154.0-installer.x86_64.iso /var/lib/libvirt/images/rhcos-43.iso
Installation of OpenShift
The installation of OpenShift starts with the creation of the Bootstrap VM. After the 3 Control Plane Nodes are online the Kube-Api-Server starts to create all required Operators on the Control Plane Nodes.
Create the Bootstrap-VM:
$virt-install --connect=qemu:///system --name bootstrap --memory 15258 --vcpus 4 --os-type=linux --os-variant=rhel7.5 --disk /var/lib/libvirt/images/bootstrap.qcow2,device=disk,bus=virtio,size=120 --disk /var/lib/libvirt/images/rhcos-43.iso,device=cdrom --network network=ocp,model=virtio,mac=52:54:00:76:34:04
Open the console in the virt-manager and start the installation. The system will boot in an emergency mode, where you can also start the boot-mode with the kernel parameter. It's a little bit more comfortable like the kernel command line
in the emergency console fill in this command:
#/usr/libexe/coreos-installer -d vda -i https://<webserver>/bootstrap.ign -b https://<webserver>/rhcos-43-metal.raw.gz -p qemu
Then unmount the ISO image and reboot the VM. Check if the VM is reachable via ssh:
$ssh core@bootstrap.test-ocp4.oc.io
Create 3 Control Plane Nodes
$virt-install --connect=qemu:///system --name master1 --memory 15258 --vcpus 4 --os-type=linux --os-variant=rhel7.5 --disk /var/lib/libvirt/images/master1.qcow2,device=disk,bus=virtio,size=120 --disk /var/lib/libvirt/images/rhcos-43.iso,device=cdrom --network network=ocp,model=virtio,mac=52:54:00:28:4e:09 --graphics=vnc --boot hd,cdrom
Again in the emergency console:
#/usr/libexe/coreos-installer
-d vda -i https://<webserver>/master.ign -b
https://<webserver>/rhcos-43-metal.raw.gz -p qemu
Repeat this for all 3 Nodes with the different MAC addresses and names.
Create 2 Worker Nodes
$virt-install --connect=qemu:///system --name worker1 --memory 15258 --vcpus 4 --os-type=linux --os-variant=rhel7.5 --disk /var/lib/libvirt/images/worker1.qcow2,device=disk,bus=virtio,size=120 --disk /var/lib/libvirt/images/rhcos-43.iso,device=cdrom --network network=ocp,model=virtio,mac=52:54:00:e4:67:14 --boot hd,cdrom
In emergency console:
#/usr/libexe/coreos-installer
-d vda -i https://<webserver>/worker.ign -b
https://<webserver>/rhcos-43-metal.raw.gz -p qemu
If you are looking to the logs on the bootstrap server have some patience... some things needs their time to bevome ready ;-)
