I recently came across Kubestack which is a GitOps framework for kubernetes Infra automation.

In my current project, we are using GCP so thought of trying out the tutorial for GKE (GCP managed kubernetes service).

The tutorial is very well documented and has clear instructions for multiple OS including Linux , macOS and Windows .
But tutorial was a bit outdated for GKE and had to make few changes for getting it to work. Also the documentation lacks the 1 click code/commands snippet “Copy” function :clipboard: .

Things that need to be updated which were not specified in documentation:

Initially i tried the macOS instructions but was blocked due to the “Internet Security” enabled on my macbook pro. Below is the error which i got:

 Error: Error accessing remote module registry
│ 
│ Failed to retrieve available versions for module "gke_zero_nginx" (gke_zero_ingress.tf:1) from kbst.xyz: Failed to request discovery document: Get
│ "https://kbst.xyz/.well-known/terraform.json": x509: certificate signed by unknown authority.

Instead of troubleshooting the macOS issue i spun up an Ubuntu, 20.04 LTS VM instance on GCP to continue the tutorial with Linux instructions.

Tutorial is categorized in 3 categories, mainly:

Local Development

# Download the latest release
curl -LO "https://github.com/kbst/kbst/releases/download/$(curl -s https://www.kubestack.com/cli-latest.txt)/kbst_linux_amd64.zip"

# Extract the binary into your PATH e.g. /usr/local/bin
sudo unzip -d /usr/local/bin/ kbst_linux_amd64.zip kbst

# Verify the binary works
kbst --version

I tried the kbst version v0.1.5

# Use the kbst CLI to scaffold your repository using the GKE starter
kbst repo init gke

# Change into the starter directory
cd kubestack-starter-gke

This repository includes all the files required for tutorial to work
Repo

and just need to update few variables that gives you control on naming and region of GCP resources
Repo

Best part of kubestack is that you can try out the framework on your local and it helps you provision local cluster by using it’s CLI:

kbst local apply

If for some reason, you get some errors and you need to start from scratch or you are done with tutorial, just do the below:

kbst local destroy

Infrastructure provisioning

Another great thing while trying this tutorial is that the kubestack team have provided a docker image preinstalled with required tools to try out this tutorial.
Run below to create the container:

# Build the bootstrap container
docker build -t kbst-infra-automation:bootstrap .

# Exec into the bootstrap container
docker run --rm -ti \
    -v `pwd`:/infra \
    kbst-infra-automation:bootstrap

When the 2nd command finishes, terminal will be switched to the container terminal.
Next up, we would need to create a GCP IAM service account which will be used in GitHub Actions.

gcloud init

Here, just follow the instructions to configure with respective gcp project.

# Use the current project
PROJECT=$(gcloud config get-value project)

# Create the service account
gcloud iam service-accounts create kubestack-automation \
  --description "SA used for Kubestack Github Actions" \
  --display-name "kubestack-automation"
  
# Assign the owner role to the service account
gcloud projects add-iam-policy-binding ${PROJECT} \
  --member serviceAccount:kubestack-automation@${PROJECT}.iam.gserviceaccount.com \
  --role roles/owner

# Create service account keys
gcloud iam service-accounts keys create \
  ~/.config/gcloud/application_default_credentials.json \
  --iam-account kubestack-automation@${PROJECT}.iam.gserviceaccount.com
  
# Activate the service account and stop using your personal account  
gcloud auth activate-service-account --key-file ~/.config/gcloud/application_default_credentials.json

As per terraform best practice, we need to save terraform state in a remote state backend . Here we will use GCP storage bucket .

# Set the location of your multi-regional bucket
# valid values are `asia`, `eu` or `us`
read -p "Bucket location: " LOCATION

# Create bucket and configure remote state
BUCKET_NAME=terraform-state-kubestack-`git rev-parse --short HEAD`
gsutil mb -l $LOCATION gs://$BUCKET_NAME
gsutil versioning set on gs://$BUCKET_NAME
cat > state.tf <<EOF
terraform {
  backend "gcs" {
    bucket = "${BUCKET_NAME}"
  }
}
EOF

Now that we have completed the bootstrap setup, we are good to run terraform commands which will provision required resources.

# Initialize Terraform
terraform init

# Create apps and ops workspaces
terraform workspace new apps
terraform workspace new ops

# Bootstrap the ops environment
terraform apply --auto-approve --target module.gke_zero
terraform apply --auto-approve

output will be something like:

Apply complete! Resources: 13 added, 0 changed, 0 destroyed.
# Bootstrap the apps environment
terraform workspace select apps
terraform apply --auto-approve --target module.gke_zero
terraform apply --auto-approve

output will be something like:

Apply complete! Resources: 18 added, 0 changed, 0 destroyed.

Terraform state file in bucket will have different state for each workspace:
Repo

Repo

Repo

Now we need to setup DNS: For Ops:

terraform workspace select ops
terraform state show module.gke_zero.module.cluster.google_dns_managed_zone.current[0]

output will be something like:

# module.gke_zero.module.cluster.google_dns_managed_zone.current[0]:
resource "google_dns_managed_zone" "current" {
    description   = "Managed by Terraform"
    dns_name      = "kbst-ops-europe-west3.gcp.poc."
    force_destroy = false
    id            = "projects/XXXXXXXX/managedZones/kbst-ops-europe-west3"
    labels        = {}
    name          = "kbst-ops-europe-west3"
    name_servers  = [
        "ns-cloud-c1.googledomains.com.",
        "ns-cloud-c2.googledomains.com.",
        "ns-cloud-c3.googledomains.com.",
        "ns-cloud-c4.googledomains.com.",
    ]
    project       = "XXXXXXXX"
    visibility    = "public"
}

For Apps:

terraform workspace select apps
terraform state show module.gke_zero.module.cluster.google_dns_managed_zone.current[0]

output will be something like:

# module.gke_zero.module.cluster.google_dns_managed_zone.current[0]:
resource "google_dns_managed_zone" "current" {
    description   = "Managed by Terraform"
    dns_name      = "kbst-apps-europe-west3.gcp.poc."
    force_destroy = false
    id            = "projects/XXXXXXXX/managedZones/kbst-apps-europe-west3"
    labels        = {}
    name          = "kbst-apps-europe-west3"
    name_servers  = [
        "ns-cloud-c1.googledomains.com.",
        "ns-cloud-c2.googledomains.com.",
        "ns-cloud-c3.googledomains.com.",
        "ns-cloud-c4.googledomains.com.",
    ]
    project       = "XXXXXXXX"
    visibility    = "public"
}

Now we can exit the bootstrap container

exit

and commit changes to repo

git add .
git commit -m "Add cluster configuration"

Configuring Automation

We need a git repo for gitops to work. Therefore we will need to create a repo. We can create one on github.

In ubuntu, gcloud credentials are stored in a different path. Therefore, below doesn’t work:

cat .user/.config/gcloud/application_default_credentials.json | base64 -w 0 && echo

Instead below works:

cat ~/.config/gcloud/application_default_credentials.json | base64 -w 0 && echo

Highlights of using kubestack:

References:

Hope you like this post and Keep Learning !