Install Apache Airflow on Kubernetes

How to Install Apache Airflow on Kubernetes with Helm

This guide will walk you through the process of installing Apache Airflow using Helm, the Kubernetes package manager, ensuring a smooth and efficient setup.

Prerequisites

Before we begin, ensure you have the following tools installed and configured:

• Kubernetes - Installed locally or in a public cloud provider
• kubectl - Configured to communicate with your Kubernetes cluster
• Helm - For managing Kubernetes applications like Apache Airflow
• Kubernetes Ingress controller - For accessing the installation on a url

Step 1: Add the Helm chart repository

First, add the repository containing the Apache Airflow helm chart:

helm repo add apache-airflow https://airflow.apache.org
helm repo update
helm search repo airflow

Step 2: Create a namespace (optional)

Isolating applications within Kubernetes namespaces is a good practice. To deploy Apache Airflow in its own namespace:

kubectl create namespace airflow

Step 3: Install Apache Airflow using Helm

Now, you'll install Apache Airflow. The helm chart name is "airflow":

helm install airflow apache-airflow/airflow --namespace airflow --create-namespace

Step 4: Verify the installation

After installation, verify that your Apache Airflow deployment is running smoothly:

kubectl get pods --namespace airflow
kubectl get all --namespace airflow

Step 5: Customize the Apache Airflow installation (optional)

This process may require customizing your deployment through a customized values.yaml file, which specifies configuration options.

helm install airflow apache-airflow/airflow --values values.yaml --namespace airflow

Step 6: Access Apache Airflow

Accessing Apache Airflow will depend on your chosen service configuration (LoadBalancer, NodePort, or Ingress). For Ingress access, ensure an Ingress controller is operational in your cluster.

Follow the instructions in the helm install notes, the output of Step 3.