Create a custom CDN - Content Delivery Network
Using Terraform, Ansible, Azure, Nginx and geoDNS created a distributed server network for content delivery.
Although this architecture can be used for creating anything which need to be deployed as `per user location` basis.
For example
- Content Delivery Network
- Global HTTPS Cache using Varnish
- Distributed Deployments
- And so on...
In this very blog, we are just going to see CDN
CDN - Content Delivery Network
![cdn example](/_next/image?url=https%3A%2F%2Fi.imgur.com%2Fcz0tTsI.png&w=3840&q=75)
CDN are server used for delivering content - images, videos, audios, texts and blog for naming few. These are located on every location where there is significant human population is present.
In the example image above, I have shown how can Company (or even we) place server at different location and create a logic to distribute request depending on client location to nearest server.
![nearest server example](/_next/image?url=https%3A%2F%2Fi.imgur.com%2Fr5yYY0B.png&w=1080&q=75)
What logic ?
It can be geoDNS
, geoDistance
, Weighted Shuffle
and Anycast
.
Go and read about them, Here is TL;DR
geoDNS : Every server has unique IP, we distribute request on the basis os IP, ASN, Region, Continent, Country.
See example image
geoDistance : Every server has unique IP, we distribute request on the basis nearest server, by calculating shortest distance.
Weighted Shuffle : Random order with weights - Mainly for preventing DDoS
Anycast: All the server has same IP, request goes to nearest server using BGP Protocol.
See example image
Here we means the DNS Server.
Anycast is hard to achieve, since involve working with hardware and network layer and publishing same IP for all server. Weighed Shuffle is not want we want. So we are going to use geoDNS and geoDistance.
That was a brief about CDNs, now lets talk about creating one.
Creating a Custom CDN Network.
We are gonna use ...
If you don't want to do these steps exactly similar, then you can do it your way. The core idea same.
Azure for creating Virtual Machines at 3 different locations
To do this, we are using Terraform - An Infrastructure as Code platform
Nginx for Static contents serving
To do this we will use Ansible
Gcore for DNS with Free geoDNS service.
Other DNS service providers, like Cloudflare does not have geoDNS for Free - Its only for Enterprise users.
Finally upload Contents to our Content Delivery Network
To do this we also use Ansible
Creating Virtual Machines - Our Servers
The task is to create multiple server (virtual machine) at different locations, and get unique IP Address of all.
I'm going to create 7 servers at these locations:
- "Central India"
- "Central US"
- "West Europe"
See how to do it with Terraform
The source code for this terraform infra setup will be found here
We created our Virtual Machine map
variable "vm_map" {
type = map(object({
name = string
location = string
size = string
}))
default = {
"vm1" = {
name = "centralindia"
location = "Central India"
size = "Standard_B1s"
}
"vm2" = {
name = "centralus"
location = "Central US"
size = "Standard_B1s"
}
"vm3" = {
name = "westeurope"
location = "West Europe"
size = "Standard_B1s"
}
}
}
And the main file, which created all other resources is
terraform {
required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = "~> 3.0.2"
}
}
required_version = ">= 1.8.5"
}
provider "azurerm" {
features {
resource_group {
prevent_deletion_if_contains_resources = false
}
}
}
resource "azurerm_resource_group" "custom-cdn" {
name = "custom-cdn-ResourceGroup"
location = "Central India" # location has no effect
# since resouce group is just a container for other resource
}
resource "azurerm_virtual_network" "custom-cdn" {
for_each = var.vm_map
name = "${each.value.name}-VNET"
location = each.value.location
address_space = [ "10.0.0.0/16" ]
resource_group_name = azurerm_resource_group.custom-cdn.name
}
resource "azurerm_subnet" "custom-cdn" {
for_each = var.vm_map
name = "${each.value.name}-Subnet"
resource_group_name = azurerm_resource_group.custom-cdn.name
virtual_network_name = azurerm_virtual_network.custom-cdn[each.key].name
address_prefixes = [ "10.0.1.0/24" ]
}
resource "azurerm_public_ip" "custom-cdn" {
for_each = var.vm_map
name = "${ each.value.name }-PublicIp"
location = each.value.location
resource_group_name = azurerm_resource_group.custom-cdn.name
allocation_method = "Static"
}
resource "azurerm_network_interface" "custom-cdn" {
for_each = var.vm_map
name = "${ each.value.name }-NIC"
location = each.value.location
resource_group_name = azurerm_resource_group.custom-cdn.name
ip_configuration {
name = "${each.value.name}-public"
subnet_id = azurerm_subnet.custom-cdn[each.key].id
public_ip_address_id = azurerm_public_ip.custom-cdn[each.key].id
private_ip_address_allocation = "Dynamic"
}
}
resource "azurerm_virtual_machine" "custom-cdn" {
for_each = var.vm_map
name = "${ each.value.name }-VM"
location = each.value.location
resource_group_name = azurerm_resource_group.custom-cdn.name
network_interface_ids = [ azurerm_network_interface.custom-cdn[each.key].id ]
vm_size = each.value.size
storage_image_reference {
publisher = "Canonical"
offer = "0001-com-ubuntu-server-jammy"
sku = "22_04-lts-gen2"
version = "latest"
}
storage_os_disk {
name = "${ each.value.name }-OsDisk"
caching = "ReadWrite"
create_option = "FromImage"
managed_disk_type = "Standard_LRS"
}
os_profile {
computer_name = each.value.name
admin_username = "custom-cdn"
admin_password = "Password1234!"
}
os_profile_linux_config {
disable_password_authentication = false
}
}
output "custom_cdn_public_ip" {
value = {
for vm in azurerm_public_ip.custom-cdn : vm.name => vm.ip_address
}
}
See the admin password is
Password1234!
- Change this to more secure one.
terraform apply
We will get all the IP Address.
# This will also print the output
terraform output
After creating the Virtual Machines, we will have all the IP address as.
custom_cdn_public_ip = {
"centralindia-PublicIp" = "4.213.167.247"
"centralus-PublicIp" = "40.86.90.42"
"westeurope-PublicIp" = "4.180.232.216"
}
Serving Static Contents using Nginx
The task is to login to each virtual machine and setup nginx and start serving static content - files and folders.
See how to do it with Ansible
The source code for this ansible is in infra
folder here
After we got all the IP Address of all VMs. Create a inventory.ini
file for storing all this data, which ansible take as input.
[azure_vms]
central-india location=central-india ansible_host=[HOST IP] ansible_user=custom-cdn ansible_ssh_pass=Password1234!
central-us location=central-us ansible_host=[HOST IP] ansible_user=custom-cdn ansible_ssh_pass=Password1234!
west-europe location=west-europe ansible_host=[HOST IP] ansible_user=custom-cdn ansible_ssh_pass=Password1234!
Replace [HOST IP]
with server IP address.
To install and setup nginx, create a nginx.conf
in the same directory.
server {
listen 80;
server_name {{ domain_name }};
add_header X-Server-Location {{ location }};
autoindex on;
root /home/custom-cdn/contents;
}
and now create an Ansible Playbook
file.
- name: Install NGINX on all VMs
hosts: azure_vms
become: yes
vars:
domain_name: cdn.kunals.me
tasks:
- name: Update apt cache
apt:
update_cache: yes
- name: Create a new contents directory
file:
path: /home/custom-cdn/contents
state: directory
mode: "0755"
- name: Install NGINX
apt:
name: nginx
state: present
- name: Create NGINX configuration for new domain
template:
src: nginx.conf
dest: /etc/nginx/sites-available/{{ domain_name }}.conf
notify:
- Restart NGINX
- name: Enable new NGINX site
file:
src: /etc/nginx/sites-available/{{ domain_name }}.conf
dest: /etc/nginx/sites-enabled/{{ domain_name }}.conf
state: link
notify:
- Restart NGINX
handlers:
- name: Restart NGINX
service:
name: nginx
state: restarted
and finally in the same directory. Create Ansible configuration file, ansible.cfg
[defaults]
host_key_checking = false
After these files, we are ready to run the playbook.
make sure you have
sshpass
installed on your system. Ansible require it.
Run the playbook
ansible-playbook -i inventory.ini setup_nginx.yml
This will install nginx on all server, create nginx.conf file in right place and restart nginx.
See how to do it Manually
FOR EACH SERVER
To Login we use SSH
ssh [email protected]
# custom-cdn is each machines username
# replace 10.10.10.10 to actual server ip
Create a folder where all the contents reside.
mkdir contents
# pwd
# /home/custom-cdn/contents
Install nginx and make sure its running.
Open the IP address in the browser you must see nginx page
Edit the nginx.conf
to add current user.
edit file /etc/nginx/nginx.conf
- user www-data;
+ user custom-cdn;
Restart nginx
sudo systemctl restart nginx
Lets write a configuration at /etc/nginx/sites-enabled/
Create a new file
sudo vim cdn.kunals.me
# choose any domain you want
And write this
server {
listen 80;
serve_name cdn.kunals.me;
add_header X-Server-Location centalindia; # place your sever location name here
autoindex on;
root /home/custom-cdn/contents;
}
Reload nginx
sudo nginx -s reload
This will make nginx server static files and folder in contents
directory of each machine.
Which means on hitting cdn.kunals.me
the files and folders in contents
folder will be served.
Setting DNS
Now lets point cdn.kunals.me
to each IP address of our servers. You heard it right, one domain name will point to multiple IP address using request distribution logic.
Our DNS of choice is Gcore DNS, reason?. It has free geoDNS support. Go ahead and create your account on Gcore.
Open the Gcore's DNS page
Click on Add Zone
![Add zone demo image](/_next/image?url=https%3A%2F%2Fi.imgur.com%2FEim3ePV.png&w=1080&q=75)
Add your domain nameserver to Gcore.
![Add domain name](/_next/image?url=https%3A%2F%2Fi.imgur.com%2F6bxHP3s.png&w=1920&q=75)
After your domain name is Active, lets create some A records.
Make sure you have Interface mode set to Advance
![Interface mode](/_next/image?url=https%3A%2F%2Fi.imgur.com%2FZz8bcFr.png&w=640&q=75)
Click on Add a Record Set
![new records set](/_next/image?url=https%3A%2F%2Fi.imgur.com%2F0dQNy38.png&w=750&q=75)
Add an A with name cdn.kunals.me
( replace it with your domain). And click on Geo Distance to select geoDistance preset.
Fill each IP address in the records.
![add a record](/_next/image?url=https%3A%2F%2Fi.imgur.com%2FebIygs0.png&w=1080&q=75)
After filling all the 3 records, click on Create.
Now we should able to hit cdn.kunals.me
and get the empty directory served by nginx.
Uploading content to our network.
Anything that is present in contents
directory will be served. Our task is to put items in this directory across all servers.
See how to do it with Ansible
Create a new ansible playbook called scp.yml
- name: Transfer files using scp
hosts: azure_vms
vars:
file_path: null # Path to the file you want to transfer
tasks:
- name: SCP file to VM
shell:
cmd: sshpass -p {{ ansible_ssh_pass }} scp {{ file_path }} {{ ansible_user }}@{{ ansible_host }}:/home/custom-cdn/contents
delegate_to: localhost
when running this playbook specify the file_path
variable you want to send.
ansible-playbook -i inventory.ini scp.yml -e "file_path=/home/kunal/hello.txt"
This way we will store any file in contents
folder of every server.
When doing it manually, for EVERY SERVER
We can use scp
- Secure Copy. An SSH based file transfer utility.
scp new_file.txt [email protected]:/home/custom-cdn/contents
# Replace 10.10.10.10 to actual server IP
Finally
We are able to store content to our content network, any we can access it at cdn.kunals.me
Bonus - Add a SSL Certificate to get HTTPS
See how to do it with Ansible
Create a new ansible playbook, called install_ssl_cert.yml
- name: Setup SSL Certificates
hosts: azure_vms
become: yes
vars:
domain_name: cdn.kunals.me
admin_email: [email protected]
tasks:
- name: Update apt cache
apt:
update_cache: yes
- name: Ensure Snapd is installed
apt:
name: snapd
state: present
- name: Install certbot
shell:
cmd: snap install --classic certbot
- name: Install Certificates
shell:
cmd: certbot --nginx -m {{ admin_email }} -d {{ domain_name }} --agree-tos --non-interactive
And run this playbook as
ansible-playbook -i inventory.ini install_ssl_cert.yml
When doing manually, FOR EACH SERVER
Install the certbot
Run this command to issue certificates.
sudo certbot --nginx
Done
Thank for reading it. For any suggestions email me - kunal [at] kunalsin9h [dot] com