E2: About Kubernetes Load Balancers
Todays episode is about accessing applications running on Kubernetes. We will look at how the Kubernetes services of type LoadBalancer work , when we should use them and how they integrate with load balancers of cloud providers. We also explain what an Ingress is and how it compares to services of type LoadBalancer.
We will also look take a brief look at the concept of finalizers and see how they help us avoid leaving unwanted cloud resources activ when deleting load balancers in Kubernetes.
Services in Kubernetes
Each Pod in Kubernetes has it’s own IP address. When a Pod is destroyed and recreated by Kubernetes, it’s IP changes. This is obviously a problem if we want to access an application running on a Pod. Services solve this problem for us by giving us a static IP through which we can access an application running on a Pod.
The default service type is ClusterIP. This type of service can only be accessed from inside the cluster. Services of type NodePort or LoadBalancer are accessible from outside the cluster.
Service type LoadBalancer vs Ingress
Kubernetes services of type LoadBalancer and Ingresses can at first glance seem like they are doing the same thing. So let’s take a closer look.
Service type LoadBalancer
A kubernetes LoadBalancer service is a service that can be accessed through external load balancers that are NOT in your kubernetes cluster, but exist elsewhere. They can work with your pods, assuming that your pods are externally routable. Google and AWS provide this capability natively. In terms of Amazon, this service maps directly with ELB. Kubernetes when running in AWS (EKS) can automatically provision and configure an ELB instance for each LoadBalancer service deployed.
The following shows how users can access pods and the applications running on them, on kubernetes when using a service type LoadBalancer with AWS.
When you are running kubernetes on premise you might not want to depend on a cloud provider for load balancing. In that case there are also alternatives such as metallb.
Ingress
An ingress is really just a set of rules to pass to a controller that is listening for them. An Ingress Controller. You can deploy a bunch of ingress rules, but nothing will happen unless you have a controller that can process them. A LoadBalancer service could listen for ingress rules, if it is configured to do so.
Let\s look at an example ingress ressource
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-example
spec:
rules:
- host: "foo.bar.com"
http:
paths:
- pathType: Prefix
path: "/bar"
backend:
service:
name: service1
port:
number: 80
- host: "bar.foo.com"
http:
paths:
- pathType: Prefix
path: "/foo"
backend:
service:
name: service2
port:
number: 80Each ingress rule consists of the following:
Host
Paths
Backend
The host is optional, in this example, the host is specified for both ingress rules, so the rule applies to inbound HTTP traffic where the host header is either foo.bar.com or bar.foo.com.
The paths section contains a list of paths, in the example above each ingress rule has one path in it’s list of paths. Each path is associated with a backend
The backend defines a service to which the traffic should be send. HTTP(S) requests that match the host and path of the rule are sent to the listed backend.
Let’s consider a second ingress resource and a visual representation of it.
This time our ingress consists of a total of three rules, all applying to the same host example.com. They differ by path, meaning that an Incoming request which has the host header set to example.com and a path of s1 will be forwarded to service1.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-example
spec:
rules:
- host: "example.com"
http:
paths:
- pathType: Prefix
path: "/s1"
backend:
service:
name: service1
port:
number: 80
- host: "example.com"
http:
paths:
- pathType: Prefix
path: "/s2"
backend:
service:
name: service2
port:
number: 80
- host: "example.com"
http:
paths:
- pathType: Prefix
path: "/s3"
backend:
service:
name: service3
port:
number: 80The following diagramm is a visual representation of the above ingress.
Service1-3 in the above example are typically of type ClusterIP since they are accessed from inside the cluster.
As you can see, the ingress rules define which service the incoming traffic should go to. It’s important to once more emphasize that ingress rules alone have no effect, you need a controller that executes them.
While a service of type LoadBalancer or Nodeport may be used with any portocol or port, an Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically uses a service of type NodePort or LoadBalancer.
When to use what?
If you want to directly expose a service, service type LoadBalancer is the default method. All traffic on the port you specify will be forwarded to the service. There is no filtering, no routing, etc. This means you can send almost any kind of traffic to it, like HTTP, TCP, UDP, Websockets, gRPC, or whatever.
The big downside is that each service you expose with a LoadBalancer will get its own IP address, and you have to pay for a LoadBalancer per exposed service, which can get expensive!
Ingress is probably the most powerful way to expose your services, but can also be the most complicated. There are many types of Ingress controllers, from the Google Cloud Load Balancer, Nginx, Contour, Istio, and more. There are also plugins for Ingress controllers, like the cert-manager, that can automatically provision SSL certificates for your services.
Ingress is the most useful if you want to expose multiple services under the same IP address, and these services all use the same L7 protocol (typically HTTP). You only pay for one load balancer if you are using the native GCP integration, and because Ingress is “smart” you can get a lot of features out of the box (like SSL, Auth, Routing, etc)
AWS Load Balancer Controller
Now, as we have seen before, the service type LoadBalance needs an external load balancer to function. In most cases these will be load balancers created on a cloud provider.
The exact setup of this varies depending on the cloud provider you use, assuming AWS you will need the AWS Load Balancer Controller.
This controller coordinates the creation of an ELB on AWS whenever you create a service of type LoadBalancer in your kubernetes cluster.
When you create a service of type LoadBalancer the AWS Load Balancer Controller will communicate with AWS to ensure that the corelating ELB is created there.
Finalizers for load balancers
Now, what happens when we delete the serivce type LoadBalancer? Ideally we want the actual load balancer in the cloud to be gone as well since it’s serves no purpose without the service.
Good news: that’s exactly what happens. when you have the AWS Load Balancer Controller deployed on your kubernetes cluster, you might notice that all your services of type LoadBalancer have a section added to their metadata that looks like this:
finalizer:
- service.k8s.aws/resourcesSimilarly, all ingresses in your cluster also have a finalizer section
finalizer:
- ingress.k8s.aws/resourcesThese finalizers will prevent the resource from being deleted until the AWS Load Balancer Controller is done cleaning up all related cloud resources.
The name of the finalizer, e.g. `service.k8s.aws/resources` really has no special meaning to it in kubernetes, it’s just a string that the load balancer controller understands.
If you put a finalizer such as
finalizer:
- this.is.just.a.random.string/doesnotexistThen the deletion of the resource will be prevented indefinetly, since there is no controller that understands this strings and knows what to do with it. The only way you can delete the service in this case to manually remove the finalizer from the service definition.
Closing Words
Next Weeks episode will feature a story about how AWS Load Balancer Controller and Strimzi Kafka Operator can be working against each other, troubleshooting an edge case that almost had me lose my sanity. Consider leaving your email if you want to hear the story.