June 24, 2024

Create a custom CDN - Content Delivery Network

Using Terraform, Ansible, Azure, Nginx and geoDNS created a distributed server network for content delivery.

Although this architecture can be used for creating anything which need to be deployed as `per user location` basis.

For example

Content Delivery Network
Global HTTPS Cache using Varnish
Distributed Deployments
And so on...

In this very blog, we are just going to see CDN

CDN - Content Delivery Network

CDN are server used for delivering content - images, videos, audios, texts and blog for naming few. These are located on every location where there is significant human population is present.

In the example image above, I have shown how can Company (or even we) place server at different location and create a logic to distribute request depending on client location to nearest server.

What logic ?

It can be geoDNS, geoDistance, Weighted Shuffle and Anycast.

Go and read about them, Here is TL;DR

geoDNS : Every server has unique IP, we distribute request on the basis os IP, ASN, Region, Continent, Country.
See example image
geoDistance : Every server has unique IP, we distribute request on the basis nearest server, by calculating shortest distance.
Weighted Shuffle : Random order with weights - Mainly for preventing DDoS
Anycast: All the server has same IP, request goes to nearest server using BGP Protocol.
See example image

Here we means the DNS Server.

Anycast is hard to achieve, since involve working with hardware and network layer and publishing same IP for all server. Weighed Shuffle is not want we want. So we are going to use geoDNS and geoDistance.

That was a brief about CDNs, now lets talk about creating one.

Creating a Custom CDN Network.

We are gonna use ...

If you don't want to do these steps exactly similar, then you can do it your way. The core idea same.

Azure for creating Virtual Machines at 3 different locations
To do this, we are using Terraform - An Infrastructure as Code platform
Nginx for Static contents serving
To do this we will use Ansible
Gcore for DNS with Free geoDNS service.
Other DNS service providers, like Cloudflare does not have geoDNS for Free - Its only for Enterprise users.
Finally upload Contents to our Content Delivery Network
To do this we also use Ansible

Creating Virtual Machines - Our Servers

The task is to create multiple server (virtual machine) at different locations, and get unique IP Address of all.

I'm going to create 7 servers at these locations:

"Central India"
"Central US"
"West Europe"

See how to do it with Terraform

The source code for this terraform infra setup will be found here

We created our Virtual Machine map

variables.tf

variable "vm_map" {
    type = map(object({
        name     = string
        location = string
        size     = string
    }))
 
    default = {
        "vm1" = {
            name = "centralindia"
            location = "Central India"
            size = "Standard_B1s"
        }
 
        "vm2" = {
            name = "centralus"
            location = "Central US"
            size = "Standard_B1s"
        }
 
        "vm3" = {
            name = "westeurope"
            location = "West Europe"
            size = "Standard_B1s"
        }
    }
}

And the main file, which created all other resources is

main.tf

terraform {
    required_providers {
        azurerm = {
            source = "hashicorp/azurerm"
            version = "~> 3.0.2"
        }
    }
 
    required_version = ">= 1.8.5"
}
 
provider "azurerm" {
    features {
        resource_group {
            prevent_deletion_if_contains_resources = false
        }
    }
}
 
resource "azurerm_resource_group" "custom-cdn" {
    name = "custom-cdn-ResourceGroup"
    location = "Central India" # location has no effect
    # since resouce group is just a container for other resource
}
 
resource "azurerm_virtual_network" "custom-cdn" {
    for_each = var.vm_map
 
    name = "${each.value.name}-VNET"
    location = each.value.location
    address_space = [ "10.0.0.0/16" ]
    resource_group_name = azurerm_resource_group.custom-cdn.name
}
 
resource "azurerm_subnet" "custom-cdn" {
    for_each = var.vm_map
 
    name = "${each.value.name}-Subnet"
    resource_group_name = azurerm_resource_group.custom-cdn.name
    virtual_network_name = azurerm_virtual_network.custom-cdn[each.key].name  
    address_prefixes = [ "10.0.1.0/24" ]
}
 
resource "azurerm_public_ip" "custom-cdn" {
    for_each = var.vm_map
 
    name = "${ each.value.name }-PublicIp"
    location = each.value.location
    resource_group_name = azurerm_resource_group.custom-cdn.name
    allocation_method = "Static"
}
 
resource "azurerm_network_interface" "custom-cdn" {
    for_each = var.vm_map
 
    name = "${ each.value.name }-NIC"
    location = each.value.location
    resource_group_name = azurerm_resource_group.custom-cdn.name
 
    ip_configuration {
        name = "${each.value.name}-public"
        subnet_id = azurerm_subnet.custom-cdn[each.key].id
        public_ip_address_id = azurerm_public_ip.custom-cdn[each.key].id
        private_ip_address_allocation = "Dynamic"
    }
}
 
resource "azurerm_virtual_machine" "custom-cdn" {
    for_each = var.vm_map
 
    name = "${ each.value.name }-VM"
    location = each.value.location
    resource_group_name = azurerm_resource_group.custom-cdn.name
    network_interface_ids = [ azurerm_network_interface.custom-cdn[each.key].id ]
    vm_size = each.value.size
 
    storage_image_reference {
        publisher = "Canonical"
        offer     = "0001-com-ubuntu-server-jammy"
        sku       = "22_04-lts-gen2"
        version   = "latest"
    }
 
    storage_os_disk {
        name              = "${ each.value.name }-OsDisk"
        caching           = "ReadWrite"
        create_option     = "FromImage"
        managed_disk_type = "Standard_LRS"
    }
 
    os_profile {
        computer_name  = each.value.name
        admin_username = "custom-cdn"
        admin_password = "Password1234!"
    }
 
    os_profile_linux_config {
        disable_password_authentication = false
    }
}
 
output "custom_cdn_public_ip" {
    value = {
        for vm in azurerm_public_ip.custom-cdn : vm.name => vm.ip_address
    }
}

See the admin password is Password1234! - Change this to more secure one.

terraform apply

We will get all the IP Address.

# This will also print the output
terraform output

After creating the Virtual Machines, we will have all the IP address as.

custom_cdn_public_ip = {
  "centralindia-PublicIp" = "4.213.167.247"
  "centralus-PublicIp" = "40.86.90.42"
  "westeurope-PublicIp" = "4.180.232.216"
}

Serving Static Contents using Nginx

The task is to login to each virtual machine and setup nginx and start serving static content - files and folders.

See how to do it with Ansible

The source code for this ansible is in infra folder here

After we got all the IP Address of all VMs. Create a inventory.ini file for storing all this data, which ansible take as input.

inventory.ini

[azure_vms]
central-india location=central-india ansible_host=[HOST IP] ansible_user=custom-cdn ansible_ssh_pass=Password1234!
central-us location=central-us ansible_host=[HOST IP] ansible_user=custom-cdn ansible_ssh_pass=Password1234!
west-europe location=west-europe ansible_host=[HOST IP] ansible_user=custom-cdn ansible_ssh_pass=Password1234!

Replace [HOST IP] with server IP address.

To install and setup nginx, create a nginx.conf in the same directory.

nginx.conf

server { 
    listen 80;
    server_name {{ domain_name }};
 
    add_header X-Server-Location {{ location }};
    autoindex on;
 
    root /home/custom-cdn/contents;
}

and now create an Ansible Playbook file.

setup_nginx.yml

- name: Install NGINX on all VMs
    hosts: azure_vms
    become: yes
    vars:
        domain_name: cdn.kunals.me
    tasks:
        - name: Update apt cache
        apt:
            update_cache: yes
        
        - name: Create a new contents directory
        file:
            path: /home/custom-cdn/contents
            state: directory
            mode: "0755"
 
        - name: Install NGINX
        apt:
            name: nginx
            state: present
 
        - name: Create NGINX configuration for new domain
        template:
            src: nginx.conf
            dest: /etc/nginx/sites-available/{{ domain_name }}.conf
        notify:
            - Restart NGINX
 
        - name: Enable new NGINX site
        file:
            src: /etc/nginx/sites-available/{{ domain_name }}.conf
            dest: /etc/nginx/sites-enabled/{{ domain_name }}.conf
            state: link
        notify:
            - Restart NGINX
 
    handlers:
        - name: Restart NGINX
            service:
                name: nginx
                state: restarted

and finally in the same directory. Create Ansible configuration file, ansible.cfg

ansible.cfg

[defaults]
host_key_checking = false

After these files, we are ready to run the playbook.

make sure you have sshpass installed on your system. Ansible require it.

Run the playbook

ansible-playbook -i inventory.ini setup_nginx.yml

This will install nginx on all server, create nginx.conf file in right place and restart nginx.

See how to do it Manually

FOR EACH SERVER

To Login we use SSH

ssh [email protected]
 
# custom-cdn is each machines username
# replace 10.10.10.10 to actual server ip

Create a folder where all the contents reside.

mkdir contents
 
# pwd
# /home/custom-cdn/contents

Install nginx and make sure its running.

Open the IP address in the browser you must see nginx page

Edit the nginx.conf to add current user.

edit file /etc/nginx/nginx.conf

- user www-data;
+ user custom-cdn;

Restart nginx

sudo systemctl restart nginx

Lets write a configuration at /etc/nginx/sites-enabled/

Create a new file

sudo vim cdn.kunals.me
# choose any domain you want

And write this

server {
 
    listen 80;
    serve_name cdn.kunals.me;
 
    add_header X-Server-Location centalindia; # place your sever location name here
 
    autoindex on;
    root /home/custom-cdn/contents;
}

Reload nginx

sudo nginx -s reload

This will make nginx server static files and folder in contents directory of each machine.

Which means on hitting cdn.kunals.me the files and folders in contents folder will be served.

Setting DNS

Now lets point cdn.kunals.me to each IP address of our servers. You heard it right, one domain name will point to multiple IP address using request distribution logic.

Our DNS of choice is Gcore DNS, reason?. It has free geoDNS support. Go ahead and create your account on Gcore.

Open the Gcore's DNS page

Click on Add Zone

Add your domain nameserver to Gcore.

After your domain name is Active, lets create some A records.

Make sure you have Interface mode set to Advance

Click on Add a Record Set

Add an A with name cdn.kunals.me ( replace it with your domain). And click on Geo Distance to select geoDistance preset.

Fill each IP address in the records.

After filling all the 3 records, click on Create.

Now we should able to hit cdn.kunals.me and get the empty directory served by nginx.

Uploading content to our network.

Anything that is present in contents directory will be served. Our task is to put items in this directory across all servers.

See how to do it with Ansible

Create a new ansible playbook called scp.yml

scp.yml

- name: Transfer files using scp
    hosts: azure_vms
    vars:
        file_path: null # Path to the file you want to transfer
    tasks:
        - name: SCP file to VM
        shell:
            cmd: sshpass -p {{ ansible_ssh_pass }} scp {{ file_path }} {{ ansible_user }}@{{ ansible_host }}:/home/custom-cdn/contents
        delegate_to: localhost

when running this playbook specify the file_path variable you want to send.

ansible-playbook -i inventory.ini scp.yml -e "file_path=/home/kunal/hello.txt"

This way we will store any file in contents folder of every server.

When doing it manually, for EVERY SERVER

We can use scp - Secure Copy. An SSH based file transfer utility.

scp new_file.txt [email protected]:/home/custom-cdn/contents
 
# Replace 10.10.10.10 to actual server IP

Finally

We are able to store content to our content network, any we can access it at cdn.kunals.me

Bonus - Add a SSL Certificate to get `HTTPS`

See how to do it with Ansible

Create a new ansible playbook, called install_ssl_cert.yml

- name: Setup SSL Certificates
hosts: azure_vms
become: yes
vars:
    domain_name: cdn.kunals.me
    admin_email: [email protected]
tasks:
    - name: Update apt cache
    apt: 
        update_cache: yes
 
    - name: Ensure Snapd is installed
    apt:
        name: snapd
        state: present
    
    - name: Install certbot 
    shell:
        cmd: snap install --classic certbot
    
    - name: Install Certificates
    shell:
        cmd: certbot --nginx -m {{ admin_email }} -d {{ domain_name }} --agree-tos --non-interactive

And run this playbook as

ansible-playbook -i inventory.ini  install_ssl_cert.yml

When doing manually, FOR EACH SERVER

Install the certbot

Run this command to issue certificates.

sudo certbot --nginx

Done

Thank for reading it. For any suggestions email me - kunal [at] kunalsin9h [dot] com

Create a custom CDN - Content Delivery Network

Using Terraform, Ansible, Azure, Nginx and geoDNS created a distributed server network for content delivery.

In this very blog, we are just going to see CDN

What logic ?

Creating a Custom CDN Network.

Creating Virtual Machines - Our Servers

Serving Static Contents using Nginx

Setting DNS

Uploading content to our network.

Finally

Bonus - Add a SSL Certificate to get HTTPS

Done

Bonus - Add a SSL Certificate to get `HTTPS`