Deploy Everyday

AWS Serverless Events into Knative (or anywhere!)

Wed, 29 Apr 2020 05:00:00 +0000

A lot of services inside AWS produces events to be consumed by services inside AWS, that’s one of the Serverless foundations. The classic example is executing an AWS Lambda function to resize an image after being uploaded to AWS S3. Receive the event, process the data. Event-driven. Serverless-styles.

Introducing Konnek

Konnek is this little project I’ve been working on to extract events from cloud providers – like AWS and GCP –, package them into CloudEvents and send them anywhere – including Knative.

It works by deploying a central AWS Lambda receiving events from anywhere inside AWS in a generic way, parsing those events according to the CloudEvents spec and send them via an HTTP POST to a webserver. Here is the list of AWS events supported by Konnek.

Konnek -> Knative

Consuming events emitted by Konnek into Knative can be quite simple. For the next steps, I assume you already have a Knative platform up and running with Service and Eventing installed and with a Broker named default in the default namespace. You can achieve all these steps by following the Knative Installation docs.

Here it’s how it looks like:

On Knative

First, let’s create a receiver Knative Service. It will be responsible to receive the event from Konnek.

Create a file named receiver.yaml with the following content and apply it with kubectl -f receiver.yaml.

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: konnek-receiver
spec:
  template:
    metadata: 
      annotations:
        # avoiding cold start for fun and profit
        autoscaling.knative.dev/minScale: "1"
    spec:
      containers:
      # You can find the code for the `knative-receiver` service here
      # https://github.com/konnek/knative-receiver
      - image: konnek/knative-receiver

Next, we will deploy a SinkBinding. What it does is to connect a Kubernetes (or Knative) component that wants to generate events, called the subject, to a resource that can consume events, called the sink. In our case, the receiver Service is forwarding the events to the default Broker – a central place in Knative to manage events:

Create a file named sinkbinding.yaml with the following content and apply it with kubectl -f sinkbinding.yaml.

apiVersion: sources.knative.dev/v1alpha1
kind: SinkBinding
metadata:
  name: konnek-sinkbinding
spec:
  subject:
    apiVersion: serving.knative.dev/v1
    kind: Service
    name: konnek-receiver
  sink:
    ref:
      apiVersion: eventing.knative.dev/v1beta1
      kind: Broker
      name: default

The third step is to set up a Knative Trigger. It will watch the Broker for an event with a specific type and trigger the code that will finally consume the event. In our example, we will consume an SQS event.

Create a file named trigger.yaml with the following content and apply it with kubectl -f trigger.yaml.

apiVersion: eventing.knative.dev/v1beta1
kind: Trigger
metadata:
  name: konnek-trigger-aws-sqs
spec:
  broker: default
  filter:
    attributes:
      # Here we say we want SQS events
      type: com.amazon.sqs
  subscriber:
    ref:
      # And trigger the consumer function
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: konnek-consumer

Finally, let’s deploy the Knative consumer Service, which will log the event in the logs – but we could do anything with it!

Create a file named consumer.yaml with the following content and apply it with kubectl -f consumer.yaml.

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: konnek-consumer
  labels:
    serving.knative.dev/visibility: cluster-local
spec:
  template:
    spec:
      containers:
      - image: konnek/consumer

Before deploying the Konnek function in AWS, we need the address from the receiver Knative Service – since Konnek will forward the AWS events there. Let’s fetch it and add into the KONNEK_CONSUMER environment variable.

export KONNEK_CONSUMER=$(kubectl get ksvc konnek-receiver -o jsonpath="{.status.url}")

On AWS

The setup is way simpler in AWS. We will use the Serverless Framework to deploy the function, so make sure it is installed and configured.

First, get the latest version of konnek-aws. For now, it’s v0.0.3, but you can check the latest version here:

wget https://github.com/konnek/konnek-aws/releases/download/v0.0.3/konnek-aws-v0.0.3.zip -O konnek.zip

Get the official Konnek serverless.yml file:

wget https://raw.githubusercontent.com/konnek/konnek-aws/master/config/serverless-framework/serverless.yml

Make sure the KONNEK_CONSUMER environment variable is set to the Knative receiver Service and deploy the function!

export KONNEK_CONSUMER=$(kubectl get ksvc konnek-receiver -o jsonpath="{.status.url}")
serverless deploy

That should be it! Let’s give it a spin!

First, start looking into the Knative consumer logs with stern, since the event will end up there:

stern -l serving.knative.dev/service=konnek-consumer -c user-container

In another terminal, get a SQS mock data:

wget https://raw.githubusercontent.com/konnek/konnek-aws/master/testdata/sqs.json

And invoke our Konnek function with the Serverless Framework invoke command using the mock data:

serverless invoke -f konnek -p sqs.json

Look in the stern terminal, you should see something like:

... user-container 2020/04/25 18:35:50 Validation: valid
... user-container Context Attributes,
... user-container   specversion: 1.0
... user-container   type: com.amazon.sqs
... user-container   source: arn:aws:sqs:eu-central-1:123456789012:MyQueue
... user-container   id: 1b6a181f-42bb-40e0-a95a-b54baf2795f0
... user-container   time: 2020-04-25T18:35:50.821658085Z
... user-container   datacontenttype: application/json
... user-container Extensions,
... user-container   knativearrivaltime: 2020-04-25T18:35:50.826800825Z
... user-container   knativehistory: default-kne-trigger-kn-channel.default.svc.cluster.local
... user-container   traceparent: 00-1fdfa2325a20d35b622a8ad2262566ff-d8445a2b188ba72f-00
... user-container Data,
... user-container   {
... user-container     "Records": [
... user-container       {
... user-container         "attributes": {
... user-container           "ApproximateFirstReceiveTimestamp": "1523232000001",
... user-container           "ApproximateReceiveCount": "1",
... user-container           "SenderId": "123456789012",
... user-container           "SentTimestamp": "1523232000000"
... user-container         },
... user-container         "awsRegion": "eu-central-1",
... user-container         "body": "Hello from SQS!",
... user-container         "eventSource": "aws:sqs",
... user-container         "eventSourceARN": "arn:aws:sqs:eu-central-1:123456789012:MyQueue",
... user-container         "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
... user-container         "messageAttributes": {},
... user-container         "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
... user-container         "receiptHandle": "MessageReceiptHandle"
... user-container       }
... user-container     ]
... user-container   }

WE DID IT! An AWS event directly into your Knative infrastructure :o

To receive events from a real SQS queue, just add its AWS ARN in the serverless.yaml file, it will look like this:

...

functions:
  konnek:
    handler: main
    events:
      - sqs: arn:aws:sqs:us-east-1:123456789012:queue1

And redeploy your function :)

What can I do with it?

We’ve just opened the AWS Serverless world to Knative! Take a look at the source code of the consumer Knative Service:

package main

import (
	"context"
	"log"

	cloudevents "github.com/cloudevents/sdk-go"
)

func display(event cloudevents.Event) {
	log.Print(event.String())
}

func main() {
	client, err := cloudevents.NewDefaultClient()
	if err != nil {
		log.Fatalf("could not create client: %v\n", err)
	}

	ctx := context.Background()

	log.Printf("server started on port %d", 8080)

	err = client.StartReceiver(ctx, display)
	if err != nil {
		log.Fatalf("failed to start server, %v", err)
	}
}

In this case, the display function is printing the function to the stdout, but you can implement whatever you want! Add the event to a database, send a message to Slack, resize images and all the cool Serverless things ⚡️

Knative (or anywhere!)

Today we saw an example to receive the events in the Knative platform, but Konnek can send them to any webserver that understands CloudEvents 😮 Try it out receiving in your local machine!.

Resources From 'The State of Open Source' Talk

Wed, 22 Apr 2020 05:00:00 +0000

Here is a bunch of links where you can find more information about the topics I covered.

They are ordered by being mentioned in the talk. Also, sometimes they are mentioned more than once, but the links are unique.

Terraform for a Highly Available VPN between AWS and Azure

Mon, 13 Apr 2020 05:00:00 +0000

Hi there 👋

Today we will go through a bunch of Terraform code to deploy a highly available site-to-site VPN between AWS and Azure, aiming for a single terraform apply execution. In the end, we will have a tunnel between an AWS VPC and an Azure vNet, reaching resources from each cloud provider as if we were in the same local network. Think about pinging an AWS EC2 from an Azure Virtual Machine, without using any public IP.

Considerations

A quick list of things to be aware of:

This is not a comprehensive Terraform tutorial, but I’ll try to explain as much as possible ;) the Learn Terraform is all you need to follow along
For simplicity, I won’t focus on Terraform remote state, modules or proper access credentials configuration, but they can all be implemented later
The setup is highly available: enough resources will be created to avoid any disruption during a possible cloud provider maintenance
A vanilla AWS and Azure account will be used to make sure we don’t mess with already running environments

We will follow a step by step process to establish the site-to-site VPN and, later, Terraform everything it with only one command!

The Diagram ™️

A lot of resources are involved in setting up a VPN between the two cloud providers. The following ~~awfully draw~~ diagram will help you visualize how the components are connected. They also have almost the same names from the Terraform code. If you feel a bit lost in the middle of the code, come back here and take a quick look 😉

Step 1 – Laying Out Azure’s Network

In the first step we will create the basic Azure infrastructure:

One Resource Group
One vNet, with the 10.0.0.0/16 network CIDR
A subnet for the virtual machines named subnet_1 and a subnet for the VPN tunnel with a mandatory name of GatewaySubnet

Create a file named azure.tf and add the following content. We will be appending more code in the next steps.

provider "azurerm" {
  version = "2.1.0"
  features {}

  # IMPORTANT!
  # For simplicity, we are not setting up "proper" access through environment variables
  # Insert your access credentials here
  subscription_id = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
  tenant_id       = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
  client_secret   = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
  client_id       = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}

# And not configuring a Terraform remote state
terraform {}

resource "azurerm_resource_group" "resource_group" {
  name     = "resource_group"
  location = "westeurope"
}

resource "azurerm_virtual_network" "vnet" {
  name                = "vnet"
  location            = azurerm_resource_group.resource_group.location
  resource_group_name = azurerm_resource_group.resource_group.name
  address_space       = ["10.0.0.0/16"]
}

# The subnet where the Virtual Machine will live
resource "azurerm_subnet" "subnet_1" {
  name                 = "subnet_1"
  resource_group_name  = azurerm_resource_group.resource_group.name
  virtual_network_name = azurerm_virtual_network.vnet.name
  address_prefix       = "10.0.1.0/24"
}

# The subnet where the VPN tunnel will live
resource "azurerm_subnet" "subnet_gateway" {
  # The name "GatewaySubnet" is mandatory
  # Only one "GatewaySubnet" is allowed per vNet
  name                 = "GatewaySubnet"
  resource_group_name  = azurerm_resource_group.resource_group.name
  virtual_network_name = azurerm_virtual_network.vnet.name
  address_prefix       = "10.0.2.0/24"
}

Go for a terraform init and a terraform apply:

$ terraform init
... ok

$ terraform apply

...
Plan: 4 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: 

Enter yes and let it rip.

Step 2 – Azure Virtual Gateway

The second step is about setting up a Virtual Network Gateway in Azure. It’s the component that understands VPN and exposes one or two public IP addresses for the connection with the other side of the tunnel (in our case, AWS). We will configure two public IPs for high availability.

In your azure.tf file, add the following:

resource "azurerm_public_ip" "public_ip_1" {
  name                = "virtual_network_gateway_public_ip_1"
  location            = azurerm_resource_group.resource_group.location
  resource_group_name = azurerm_resource_group.resource_group.name

  # Public IP needs to be dynamic for the Virtual Network Gateway
  # Keep in mind that the IP address will be "dynamically generated" after
  # being attached to the Virtual Network Gateway below
  allocation_method = "Dynamic"
}

resource "azurerm_public_ip" "public_ip_2" {
  name                = "virtual_network_gateway_public_ip_2"
  location            = azurerm_resource_group.resource_group.location
  resource_group_name = azurerm_resource_group.resource_group.name

  # Public IP needs to be dynamic for the Virtual Network Gateway
  allocation_method = "Dynamic"
}

resource "azurerm_virtual_network_gateway" "virtual_network_gateway" {
  name                = "virtual_network_gateway"
  location            = azurerm_resource_group.resource_group.location
  resource_group_name = azurerm_resource_group.resource_group.name

  type     = "Vpn"
  vpn_type = "RouteBased"

  # Configuration for high availability
  active_active = true
  # This might me expensive, check the prices  
  sku           = "VpnGw1"

  # Configuring the two previously created public IP Addresses
  ip_configuration {
    name                          = azurerm_public_ip.public_ip_1.name
    public_ip_address_id          = azurerm_public_ip.public_ip_1.id
    private_ip_address_allocation = "Dynamic"
    subnet_id                     = azurerm_subnet.subnet_gateway.id
  }

  ip_configuration {
    name                          = azurerm_public_ip.public_ip_2.name
    public_ip_address_id          = azurerm_public_ip.public_ip_2.id
    private_ip_address_allocation = "Dynamic"
    subnet_id                     = azurerm_subnet.subnet_gateway.id
  }
}

Apply your changes:

$ terraform apply
...


Plan: 3 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

Buckle up: it takes up to 30 MINUTES to create a Virtual Network Gateway. No kidding:

azurerm_virtual_network_gateway.virtual_network_gateway: Still creating... [24m51s elapsed]
azurerm_virtual_network_gateway.virtual_network_gateway: Creation complete after 24m55s [id=/subscriptions/xxx/resourceGroups/resource_group/providers/Microsoft.Network/virtualNetworkGateways/virtual_network_gateway]

Step 3 – AWS Basics

Let’s switch to AWS for a bit. Let’s create a basic AWS network infrastructure from scratch, including:

VPC
Subnet
Internet Gateway
Route Table & Route

We are not using the default AWS network infrastructure to avoid changing anything already deployed ;)

Create a new file named aws.tf and insert the code below:

provider "aws" {
  version    = "2.55.0"
  # IMPORTANT!
  # Setup your correct region, access_key and secret_key
  # Again, we are not focusing on credentials best practices here
  region     = "eu-west-1"
  access_key = ""
  secret_key = ""
}

resource "aws_vpc" "vpc" {
  cidr_block = "192.168.0.0/16"

  tags = {
    Name = "vpc"
  }
}

# The subnet where the Virtual Machine will live
resource "aws_subnet" "subnet_1" {
  vpc_id     = aws_vpc.vpc.id
  cidr_block = "192.168.1.0/24"

  tags = {
    Name = "subnet_1"
  }
}

resource "aws_internet_gateway" "internet_gateway" {
  vpc_id = aws_vpc.vpc.id

  tags = {
    Name = "internet_gateway"
  }
}

resource "aws_route_table" "route_table" {
  vpc_id = aws_vpc.vpc.id

  tags = {
    Name = "route_table"
  }
}

# Enabling the resources from subnet_1 to access the Internet
# So we can access it later via SSH
resource "aws_route" "subnet_1_exit_route" {
  route_table_id         = aws_route_table.route_table.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = aws_internet_gateway.internet_gateway.id
}

resource "aws_route_table_association" "route_table_association" {
  subnet_id      = aws_subnet.subnet_1.id
  route_table_id = aws_route_table.route_table.id
}

Apply tha changes:

$ terraform apply
...

Plan: 6 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

Getting there…

Step 4 – AWS VPN stuff

I’m very proud you made to Step 4 in this adventure, we are almost there! Let’s get some VPN goodies from AWS:

One VPN Gateway
Two Customer Gateways for HA
Two VPN Connections: this piece has the external IP addresses used for the connection with Azure
Two VPN Connection Routes to tell AWS about the Azure’s vNet CIDR
One entry in the Route Table to teach AWS how to get to Azure

We are also using Terraform’s data resource to fetch the two public IP addresses from AWS. Why? Because those IP addresses are Dynamic and generated only after the Azure’s Virtual Network Gateway is up and running. Also, you will see the azurerm_public_ip data resource has an interpolated name. Let me explain.

To achieve one terraform apply command we need to deal with inter-cloud dependencies. The critical one is getting the Azure’s public IP addresses – which are generated only after the Virtual Network Gateway is created – to be used in the AWS side. Using the interpolation in the name argument forces the data resource to run after the Virtual Network Gateway is created.

But wait. You could use depends_on on the data resource, right? Yes and no. It works on the first run, but since the depends_on forces the read of the data source to always happen in the apply phase, it triggers changes in the AWS components depending on in. The configuration never converges.

Ayyy, append the following code in your aws.tf file:

data "azurerm_public_ip" "azure_public_ip_1" {
  # That neat name interpolation
  # The end result is _exactly_ the name of Azure's public IP
  name                = "${azurerm_virtual_network_gateway.virtual_network_gateway.name}_public_ip_1"
  resource_group_name = azurerm_resource_group.resource_group.name
}

data "azurerm_public_ip" "azure_public_ip_2" {
  name                = "${azurerm_virtual_network_gateway.virtual_network_gateway.name}_public_ip_2"
  resource_group_name = azurerm_resource_group.resource_group.name
}

resource "aws_customer_gateway" "customer_gateway_1" {
  bgp_asn = 65000

  # Using the previously fetched Azure's public IP
  ip_address = data.azurerm_public_ip.azure_public_ip_1.ip_address
  type       = "ipsec.1"

  tags = {
    Name = "customer_gateway_1"
  }
}

resource "aws_customer_gateway" "customer_gateway_2" {
  bgp_asn = 65000

  ip_address = data.azurerm_public_ip.azure_public_ip_2.ip_address
  type       = "ipsec.1"

  tags = {
    Name = "customer_gateway_2"
  }
}

resource "aws_vpn_gateway" "vpn_gateway" {
  vpc_id = aws_vpc.vpc.id

  tags = {
    Name = "vpn_gateway"
  }
}

# We will use information from this piece to finish the Azure configuration on the next Step
resource "aws_vpn_connection" "vpn_connection_1" {
  vpn_gateway_id      = aws_vpn_gateway.vpn_gateway.id
  customer_gateway_id = aws_customer_gateway.customer_gateway_1.id
  type                = "ipsec.1"
  static_routes_only  = true

  tags = {
    Name = "vpn_connection_1"
  }
}

# We will use information from this piece to finish the Azure configuration on the next Step
resource "aws_vpn_connection" "vpn_connection_2" {
  vpn_gateway_id      = aws_vpn_gateway.vpn_gateway.id
  customer_gateway_id = aws_customer_gateway.customer_gateway_2.id
  type                = "ipsec.1"
  static_routes_only  = true

  tags = {
    Name = "vpn_connection_2"
  }
}

resource "aws_vpn_connection_route" "vpn_connection_route_1" {
  # Azure's vnet CIDR
  destination_cidr_block = azurerm_virtual_network.vnet.address_space[0]
  vpn_connection_id      = aws_vpn_connection.vpn_connection_1.id
}

resource "aws_vpn_connection_route" "vpn_connection_route_2" {
  # Azure's vnet CIDR
  destination_cidr_block = azurerm_virtual_network.vnet.address_space[0]
  vpn_connection_id      = aws_vpn_connection.vpn_connection_2.id
}

# The route teaching where to go to get to Azure's CIDR
resource "aws_route" "route_to_azure" {
  route_table_id = aws_route_table.route_table.id

  # Azure's vnet CIDR
  destination_cidr_block = azurerm_virtual_network.vnet.address_space[0]
  gateway_id             = aws_vpn_gateway.vpn_gateway.id
}

Do it! Nothing is impossible:

$ terraform apply
...


Plan: 8 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

It takes about 7 minutes to finish. Definitely not bad as Azure 😅

Step 5 – Establishing the Connection on Azure

The last Step created all the AWS resources needed for the VPN. The final piece of the puzzle is to create the Azure components responsible for connecting to AWS, all those arrows in the diagram.

Be mindful that each AWS VPN Connection created has two tunnels with IP addresses and secret keys. To accomplish full high availability, we need to point each Azure Virtual Network Gateway IP address to two IPs on AWS, creating a connection mesh.

The last VPN deploy includes (I know, it is a lot!):

4 Local Network Gateways, one for each AWS VPN tunnel
4 Virtual Network Gateway Connection, one for each AWS VPN tunnel

Back to the azure.tf file, add:

# Tunnel from Azure to AWS vpn_connection_1 (tunnel1)
resource "azurerm_local_network_gateway" "local_network_gateway_1_tunnel1" {
  name                = "local_network_gateway_1_tunnel1"
  location            = azurerm_resource_group.resource_group.location
  resource_group_name = azurerm_resource_group.resource_group.name

  # AWS VPN Connection public IP address
  gateway_address = aws_vpn_connection.vpn_connection_1.tunnel1_address

  address_space = [
    # AWS VPC CIDR
    aws_vpc.vpc.cidr_block
  ]
}

resource "azurerm_virtual_network_gateway_connection" "virtual_network_gateway_connection_1_tunnel1" {
  name                = "virtual_network_gateway_connection_1_tunnel1"
  location            = azurerm_resource_group.resource_group.location
  resource_group_name = azurerm_resource_group.resource_group.name

  type                       = "IPsec"
  virtual_network_gateway_id = azurerm_virtual_network_gateway.virtual_network_gateway.id
  local_network_gateway_id   = azurerm_local_network_gateway.local_network_gateway_1_tunnel1.id

  # AWS VPN Connection secret shared key
  shared_key = aws_vpn_connection.vpn_connection_1.tunnel1_preshared_key
}

# Tunnel from Azure to AWS vpn_connection_1 (tunnel2)
resource "azurerm_local_network_gateway" "local_network_gateway_1_tunnel2" {
  name                = "local_network_gateway_1_tunnel2"
  location            = azurerm_resource_group.resource_group.location
  resource_group_name = azurerm_resource_group.resource_group.name

  gateway_address = aws_vpn_connection.vpn_connection_1.tunnel2_address

  address_space = [
    aws_vpc.vpc.cidr_block
  ]
}

resource "azurerm_virtual_network_gateway_connection" "virtual_network_gateway_connection_1_tunnel2" {
  name                = "virtual_network_gateway_connection_1_tunnel2"
  location            = azurerm_resource_group.resource_group.location
  resource_group_name = azurerm_resource_group.resource_group.name

  type                       = "IPsec"
  virtual_network_gateway_id = azurerm_virtual_network_gateway.virtual_network_gateway.id
  local_network_gateway_id   = azurerm_local_network_gateway.local_network_gateway_1_tunnel2.id

  shared_key = aws_vpn_connection.vpn_connection_1.tunnel2_preshared_key
}

# Tunnel from Azure to AWS vpn_connection_2 (tunnel1)
resource "azurerm_local_network_gateway" "local_network_gateway_2_tunnel1" {
  name                = "local_network_gateway_2_tunnel1"
  location            = azurerm_resource_group.resource_group.location
  resource_group_name = azurerm_resource_group.resource_group.name

  gateway_address = aws_vpn_connection.vpn_connection_2.tunnel1_address

  address_space = [
    aws_vpc.vpc.cidr_block
  ]
}

resource "azurerm_virtual_network_gateway_connection" "virtual_network_gateway_connection_2_tunnel1" {
  name                = "virtual_network_gateway_connection_2_tunnel1"
  location            = azurerm_resource_group.resource_group.location
  resource_group_name = azurerm_resource_group.resource_group.name

  type                       = "IPsec"
  virtual_network_gateway_id = azurerm_virtual_network_gateway.virtual_network_gateway.id
  local_network_gateway_id   = azurerm_local_network_gateway.local_network_gateway_2_tunnel1.id

  shared_key = aws_vpn_connection.vpn_connection_2.tunnel1_preshared_key
}

# Tunnel from Azure to AWS vpn_connection_2 (tunnel2)
resource "azurerm_local_network_gateway" "local_network_gateway_2_tunnel2" {
  name                = "local_network_gateway_2_tunnel2"
  location            = azurerm_resource_group.resource_group.location
  resource_group_name = azurerm_resource_group.resource_group.name

  gateway_address = aws_vpn_connection.vpn_connection_2.tunnel2_address

  address_space = [
    aws_vpc.vpc.cidr_block
  ]
}

resource "azurerm_virtual_network_gateway_connection" "virtual_network_gateway_connection_2_tunnel2" {
  name                = "virtual_network_gateway_connection_2_tunnel2"
  location            = azurerm_resource_group.resource_group.location
  resource_group_name = azurerm_resource_group.resource_group.name

  type                       = "IPsec"
  virtual_network_gateway_id = azurerm_virtual_network_gateway.virtual_network_gateway.id
  local_network_gateway_id   = azurerm_local_network_gateway.local_network_gateway_2_tunnel2.id

  shared_key = aws_vpn_connection.vpn_connection_2.tunnel2_preshared_key
}

And apply it:

$ terraform apply

Plan: 8 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

Wait for it…

WE DID IT

Let the connection sit a bit to get itself together. After a minute or two, inspect the connection in the dashboard.

On AWS, go to VPC > Site-to-Site VPN Connections and take a look at the Tunnel Details tab under vpn_connection_1 and vpn_connection_2:

The green color feels very good, doesn’t it?

On Azure, go to Virtual Network Gateways > Connections and this beauty will pop up:

To make sure the configuration is converged, just run another terraform apply:

$ terraform apply

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

YES, WE DID IT! 🎉🎊🍾

The Final Test

A VPN tunnel is beautiful, but we need to make sure it works. Let’s create one Virtual Machine in each side and try to ping them through their local networks – no public IP pinging allowed here.

Create a new file named vms.tf and throw the following code there:

### Azure
resource "azurerm_public_ip" "public_ip_vm" {
  name                = "public_ip_vm"
  location            = azurerm_resource_group.resource_group.location
  resource_group_name = azurerm_resource_group.resource_group.name

  allocation_method = "Static"
}

resource "azurerm_network_interface" "network_interface_vm" {
  name                = "network_interface_vm"
  location            = azurerm_resource_group.resource_group.location
  resource_group_name = azurerm_resource_group.resource_group.name

  ip_configuration {
    name                          = "internal"
    subnet_id                     = azurerm_subnet.subnet_1.id
    private_ip_address_allocation = "Dynamic"
    public_ip_address_id = azurerm_public_ip.public_ip_vm.id
  }
}

resource "azurerm_linux_virtual_machine" "vm" {
  name                = "vm"
  location            = azurerm_resource_group.resource_group.location
  resource_group_name = azurerm_resource_group.resource_group.name
  size                = "Standard_F2"
  admin_username      = "ubuntu"

  network_interface_ids = [
    azurerm_network_interface.network_interface_vm.id,
  ]

  os_disk {
    caching              = "ReadWrite"
    storage_account_type = "Standard_LRS"
  }

  admin_ssh_key {
    username   = "ubuntu"
    # IMPORTANT
    # Add here your own public SSH key, so we can access the VM
    public_key = ""
  }

  source_image_reference {
    publisher = "Canonical"
    offer     = "UbuntuServer"
    sku       = "18.04-LTS"
    version   = "latest"
  }
}

output "azure_vm_public_ip" {
  value = azurerm_linux_virtual_machine.vm.public_ip_address
}

output "azure_vm_private_ip" {
  value = azurerm_linux_virtual_machine.vm.private_ip_address
}

### AWS
resource "aws_security_group" "ssh" {
  vpc_id = aws_vpc.vpc.id

  ingress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "security_group_ssh"
  }
}

resource "aws_key_pair" "ssh_key" {
  key_name   = "ssh_key"
  # IMPORTANT
  # Add here your own public SSH key, so we can access the VM
  public_key = ""
}

resource "aws_instance" "vm" {
  ami           = "ami-0701e7be9b2a77600"
  instance_type = "t2.micro"

  vpc_security_group_ids      = [aws_security_group.ssh.id]
  subnet_id                   = aws_subnet.subnet_1.id
  associate_public_ip_address = true

  key_name = aws_key_pair.ssh_key.key_name
}

output "aws_vm_public_ip" {
  value = aws_instance.vm.public_ip
}

output "aws_vm_private_ip" {
  value = aws_instance.vm.private_ip
}

And, for the last time, let it rip:

$ terraform apply
...

Plan: 6 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes
...

Apply complete! Resources: 6 added, 0 changed, 0 destroyed.

Outputs:

aws_vm_private_ip = 192.168.1.218
aws_vm_public_ip = 34.251.101.219
azure_vm_private_ip = 10.0.1.4
azure_vm_public_ip = 52.166.20.226

With the Terraform output, SSH into each machine with the public_ip and try to ping one another through the private_ip (with your own output data):

From AWS:

$ ssh ubuntu@34.251.101.219 ping 10.0.1.4 -c 4

PING 10.0.1.4 (10.0.1.4) 56(84) bytes of data.
64 bytes from 10.0.1.4: icmp_seq=1 ttl=64 time=21.3 ms
64 bytes from 10.0.1.4: icmp_seq=2 ttl=64 time=20.7 ms
64 bytes from 10.0.1.4: icmp_seq=3 ttl=64 time=21.3 ms
64 bytes from 10.0.1.4: icmp_seq=4 ttl=64 time=22.3 ms

--- 10.0.1.4 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 20.761/21.454/22.353/0.591 ms

From Azure:

$ ssh ubuntu@52.166.20.226 ping 192.168.1.218 -c 4

PING 192.168.1.218 (192.168.1.218) 56(84) bytes of data.
64 bytes from 192.168.1.218: icmp_seq=1 ttl=64 time=20.1 ms
64 bytes from 192.168.1.218: icmp_seq=2 ttl=64 time=20.2 ms
64 bytes from 192.168.1.218: icmp_seq=3 ttl=64 time=20.4 ms
64 bytes from 192.168.1.218: icmp_seq=4 ttl=64 time=20.5 ms

--- 192.168.1.218 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 20.119/20.338/20.539/0.169 ms

Awesome, right?

Bonus – One Apply to Rule them All

A step-by-step process is amazing to learn, but in real life, we should strive to layout our Terraform code in a way that one terraform apply brings everything up, especially when we are creating new environments. Everything so far was based on that philosophy.

For the final proof, let’s destroy the infrastructure and bring it back up.

🚨 If you are running this code in an established running environment – be it dev, staging or production – make sure you know what is being destroyed and that is safe to proceed!

$ terraform destroy
...

Plan: 0 to add, 0 to change, 35 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

And, let’s recreate everything. It will take a lot of time, let it run in a separate terminal and go pet a dog.

$ terraform apply

Plan: 35 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

...

Apply complete! Resources: 35 added, 0 changed, 0 destroyed.

Outputs:

aws_vm_private_ip = 192.168.1.83
aws_vm_public_ip = 34.252.37.174
azure_vm_private_ip = 10.0.1.4
azure_vm_public_ip = 137.117.140.139

Do the ping test again and voilà. You are the best.

aaaaaaaaaaaaaaaaaaand, scene!

Well, that was a lot, but we did it! An enormous amount of Terraform code and our highly available site-to-site VPN between AWS and Azure is up and running! If you wanna dig deep, explore the documentation about each component to clearly understand their role in the infrastructure.

All the feedback is appreciated ✨

Three Solutions to Run Knative Locally

Tue, 31 Mar 2020 05:00:00 +0000

Knative is a set of building blocks to run Serverless workloads on top of Kubernetes. You can also use those building blocks to build your own Serverless platform.

One problem with Knative is running it locally. Kubernetes is quite heavy, running yet another platform on top of it is a killer. Today we will take a look at some different approaches to test Knative without burning down your computer.

Some notes before proceeding:

I’ll be focusing on the Serving part of Knative. Eventing will probably work just fine – you just need more resources
I choose Kourier for the Knative network layer – it’s simpler and lighter than Istio
Ubuntu was used for everything, minus the Docker on Mac part ;)
You can find the full instructions to install Knative here

k3s

k3s is a lite version of Kubernetes, made for IoT & Edge computing – perfect for your laptop. k3s is packaged as a single binary (!!!), making it very easy to run. It only works on Linux :( Mac and Windows users will need a virtual machine – take a look at multipass or Digital Ocean.

Let’s do it!

export KNATIVE_VERSION="0.13.0"
export KUBECONFIG="/var/lib/rancher/k3s/server/cred/admin.kubeconfig"

# Since Knative has its own network layer, we need to disable k3s' Traefik during its installation
# to make sure Kourier proxy gets a LoadBalancer IP
curl -sfL https://get.k3s.io | sh -s - --disable traefik

# Install Knative Serving
kubectl apply --filename "https://github.com/knative/serving/releases/download/v$KNATIVE_VERSION/serving-crds.yaml"
kubectl apply --filename "https://github.com/knative/serving/releases/download/v$KNATIVE_VERSION/serving-core.yaml"

# Configure the magic xip.io DNS name
kubectl apply --filename "https://github.com/knative/serving/releases/download/v$KNATIVE_VERSION/serving-default-domain.yaml"

# Install and configure Kourier
kubectl apply --filename https://raw.githubusercontent.com/knative/serving/v$KNATIVE_VERSION/third_party/kourier-latest/kourier.yaml
kubectl patch configmap/config-network --namespace knative-serving --type merge --patch '{"data":{"ingress.class":"kourier.ingress.networking.knative.dev"}}'

Alright, the installation is done! Make sure all the Pods are running (if not, wait a bit):

kubectl get pod -A

And that k3s gave Kourier a LoadBalancer IP address:

kubectl get svc kourier -n kourier-system

                                       # yay!
NAME      TYPE           CLUSTER-IP    EXTERNAL-IP       PORT(S)                      AGE
kourier   LoadBalancer   10.43.94.13   172.17.255.1      80:31619/TCP,443:30649/TCP   80s

Let’s deploy the Knative Service sample:

kubectl apply -f https://gist.githubusercontent.com/jonatasbaldin/bc04de2e376be23f75bb5815041fdd61/raw/d2345ac9aa01d0f3c771e9b3d4a1421dd766e0f9/service.yaml

service.serving.knative.dev/helloworld-go created

Make sure it is running and find its “public” address:

kubectl get ksvc

NAME            URL                                                   LATESTCREATED         LATESTREADY           READY   REASON
helloworld-go   http://helloworld-go.default.172.17.255.1.xip.io      helloworld-go-2xvpx   helloworld-go-2xvpx   True    

Finally:

curl http://helloworld-go.default.172.17.255.1.xip.io

Hello Go Sample v1!

✨🎉🍾

✨BONUS ✨

If you run the same commands in a Digital Ocean droplet, you will get a public IP address! Quick and easy to test a web facing Service:

kubectl get ksvc

NAME                   URL                                                       LATESTCREATED                LATESTREADY                  READY   REASON
helloworld-go          http://helloworld-go.default.<my-public-ip>.xip.io        helloworld-go-7ejj4          helloworld-go-7ejj4          True    

kind

kind runs Kubernetes inside a Docker container. As crazy as it seems, it works amazingly for testing and local development with a small resource footprint.

kind do works on Docker for Mac, but getting the LoadBalancer is not trivial :( With Linux, we can get a LB using MetalLB – I used this incredible tutorial to learn about it. Again, you can totes use a virtual machine here.

Let’s start with kind and Knative:

# Getting kind
curl -Lo kind https://github.com/kubernetes-sigs/kind/releases/download/v0.7.0/kind-$(uname)-amd64
chmod +x kind
mv kind /usr/local/bin

# Getting kubectl
curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
chmod +x kubectl
mv kubectl /usr/local/bin

# Installing Docker (just copied from the Docker docs)
apt-get update
apt-get install apt-transport-https ca-certificates  curl gnupg-agent software-properties-common -y 
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
apt-get update
apt-get install docker-ce docker-ce-cli containerd.io -y

# Create a kind cluster
kind create cluster

# Install the Knative Serving
export KNATIVE_VERSION="0.13.0"
kubectl apply --filename "https://github.com/knative/serving/releases/download/v$KNATIVE_VERSION/serving-crds.yaml"
kubectl apply --filename "https://github.com/knative/serving/releases/download/v$KNATIVE_VERSION/serving-core.yaml"

# Configure the magic xip.io DNS name
kubectl apply --filename "https://github.com/knative/serving/releases/download/v$KNATIVE_VERSION/serving-default-domain.yaml"

# Install and configure Kourier
kubectl apply --filename https://raw.githubusercontent.com/knative/serving/v$KNATIVE_VERSION/third_party/kourier-latest/kourier.yaml
kubectl patch configmap/config-network --namespace knative-serving --type merge --patch '{"data":{"ingress.class":"kourier.ingress.networking.knative.dev"}}'

Now, some MetalLB steps:

# https://mauilion.dev/posts/kind-metallb/

# First, find the network CIDR used by the Docker bridge network
# MetalLB will use some IPs from there
docker inspect network bridge

[
    {
        "Name": "bridge",
        # ...
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    # this is what we want
                    # we will give the 10 addresses to MetalLB
                    # let's say, 172.17.255.1-172.17.255.250
                    "Subnet": "172.17.0.0/16"
                }
            ]
        },
        # ...
    }
]

Installing MetalLB:

# god bless the yaml
kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.9.3/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.9.3/manifests/metallb.yaml
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

# Creating a MetalLB configuration
# It states that we will use those IP addresses for the LoadBalancer
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      # the range from before!
      - 172.17.255.1-172.17.255.250
EOF

Alright, the installation is done! Make sure all the Pods are running (if not, wait a bit):

kubectl get pod -A

After that, take a look at Kourier:

kubectl get svc kourier -n kourier-system

                                       # yes!
NAME      TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)                      AGE
kourier   LoadBalancer   10.96.7.120   172.17.255.1   80:30201/TCP,443:30909/TCP   110s

Let’s deploy the Knative Service sample:

kubectl apply -f https://gist.githubusercontent.com/jonatasbaldin/bc04de2e376be23f75bb5815041fdd61/raw/d2345ac9aa01d0f3c771e9b3d4a1421dd766e0f9/service.yaml

service.serving.knative.dev/helloworld-go created

Make sure it is running and find its “public” address:

kubectl get ksvc

NAME                   URL                                                       LATESTCREATED                LATESTREADY                  READY   REASON
helloworld-go          http://helloworld-go.default.172.17.255.1.xip.io          helloworld-go-7sttx          helloworld-go-7sttx          True    

Finally:

curl http://helloworld-go.default.172.17.255.1.xip.io

Hello Go Sample v1!

Docker for Mac

Might be my favorite 😅 The Kubernetes shipped with Docker for Mac is a very simple way to use the platform locally: click on the button on your Docker configurations and you are good to go. Unfortunately, not that easy for Knative, but let’s fix that.

First, Knative only supports K8S v1.15.0+ and the default Docker for Mac comes with v1.14.6, without an upgrade path. BUT, you can get v1.16.0+ with the Edge release. It incorporates experimental features and, as such, might not be 100% stable. Whatever, we like living on the Edge. Read the Switch between Stable and Edge versions before proceeding to the download page.

Once installed, let’s do the Knative dance:

# Install the Knative Serving
export KNATIVE_VERSION="0.13.0"
kubectl apply --filename "https://github.com/knative/serving/releases/download/v$KNATIVE_VERSION/serving-crds.yaml"
kubectl apply --filename "https://github.com/knative/serving/releases/download/v$KNATIVE_VERSION/serving-core.yaml"

# Install and configure Kourier
kubectl apply --filename https://raw.githubusercontent.com/knative/serving/v$KNATIVE_VERSION/third_party/kourier-latest/kourier.yaml
kubectl patch configmap/config-network --namespace knative-serving --type merge --patch '{"data":{"ingress.class":"kourier.ingress.networking.knative.dev"}}'

If we inspect the Kourier LoadBalancer, it gets a localhost address 🤔

kubectl get svc kourier -n kourier-system

NAME      TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
kourier   LoadBalancer   10.110.153.164   localhost     80:31650/TCP,443:30315/TCP   34m

The LB is being exposed to our localhost, on ports 80 and 443. This is important and will affect the way we communicate with the Knative Service.

Let’s deploy the Knative Service sample:

kubectl apply -f https://gist.githubusercontent.com/jonatasbaldin/bc04de2e376be23f75bb5815041fdd61/raw/d2345ac9aa01d0f3c771e9b3d4a1421dd766e0f9/service.yaml

service.serving.knative.dev/helloworld-go created

Make sure it is running and find its “public” address:

kubectl get ksvc

NAME            URL                                        LATESTCREATED         LATESTREADY           READY   REASON
helloworld-go   http://helloworld-go.default.example.com   helloworld-go-9qsq7   helloworld-go-9qsq7   True    

Differently from the other steps, we didn’t apply the Magic xip.io DNS Knative file because we don’t work when the LoadBalancer is pre-configured with a domain (which, in this case, is localhost).

Finally, to access the Service, hit the LoadBalancer on localhost:80 and pass the Service domain as the Host header:

curl localhost:80 -H "Host: helloworld-go.default.example.com"

Hello Go Sample v1!

Awesome, right!?

PS: I didn’t dive too much into this solution. The localhost LoadBalancer might have its quirks that I haven’t encountered. I’ll keep this article updated if I see anything suspicious.

minikube

The Minikube section is a contribution by Jonathan Beri, you can find his original gist here. Thanks a lot!

Minikube is perfectly described in their GitHub repository:

minikube implements a local Kubernetes cluster on macOS, Linux, and Windows. minikube’s primary goals are to be the best tool for local Kubernetes application development and to support all Kubernetes features that fit.

# Get a Minikube VM up and running, with enough resources
minikube start --cpus=4 --memory=4096 --addons=ingress

# Install the Knative Serving
export KNATIVE_VERSION="0.13.0"
kubectl apply --filename "https://github.com/knative/serving/releases/download/v$KNATIVE_VERSION/serving-crds.yaml"
kubectl apply --filename "https://github.com/knative/serving/releases/download/v$KNATIVE_VERSION/serving-core.yaml"

# Configure the magic xip.io DNS name
kubectl apply --filename "https://github.com/knative/serving/releases/download/v$KNATIVE_VERSION/serving-default-domain.yaml"

# Install and configure Kourier
kubectl apply --filename https://raw.githubusercontent.com/knative/serving/v$KNATIVE_VERSION/third_party/kourier-latest/kourier.yaml
kubectl patch configmap/config-network --namespace knative-serving --type merge --patch '{"data":{"ingress.class":"kourier.ingress.networking.knative.dev"}}'

Alright, the installation is done! Make sure all the Pods are running (if not, wait a bit):

kubectl get pod -A

Open a new terminal console and start the Minikube LoadBalancer:

minikube tunnel

Status:
        machine: minikube
        pid: 13391
        route: 10.96.0.0/12 -> 192.168.64.6
        minikube: Running
        services: [kourier]
    errors: 
                minikube: no errors
                router: no errors
                loadbalancer emulator: no errors

After that, get the public IP from Kourier:

kubectl get svc kourier -n kourier-system

                                        # yes!
NAME      TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)                      AGE
kourier   LoadBalancer   10.107.29.82   10.107.29.82   80:30354/TCP,443:30261/TCP   2m47s

Let’s deploy the Knative Service sample:

kubectl apply -f https://gist.githubusercontent.com/jonatasbaldin/bc04de2e376be23f75bb5815041fdd61/raw/d2345ac9aa01d0f3c771e9b3d4a1421dd766e0f9/service.yaml

service.serving.knative.dev/helloworld-go created

Make sure it is running and find its “public” address:

kubectl get ksvc

NAME                URL                                                    LATESTCREATED             LATESTREADY               READY   REASON
helloworld-go       http://helloworld-go.default.10.107.29.82.xip.io       helloworld-go-xqlxs       helloworld-go-xqlxs       True    

Finally:

curl http://helloworld-go.default.10.107.29.82.xip.io

Hello Go Sample v1!

✨ Bonus: inlets-operator ✨

Imagine if you could easily expose all these solutions to the public Internet? Well, inlets-operator does just that. I’ll leave you with this article for your next adventure.

Avoiding Azure Functions Downtime with Slots

Fri, 27 Mar 2020 05:00:00 +0000

This week I was building a Terraform module to deploy an Azure Functions infrastructure – including Storage Account, Service Plan and the Function App – so later the development team can simply use the func CLI to deploy they functions themselves.

In the middle of my experiments, I thought about the Function update process: what happens when I change the source code and deploy a new version?

Here I am only exploring the downtime issue. Please refer to this article to learn how to deploy Azure Functions with Terraform._

Test #01 – Terraform Does Everything ™️

In this scenario, the Function App deployed by Terraform points directly to a zip file in an Azure Container, like the code below:

resource "azurerm_function_app" "fn_app" {
  name                      = "my-name"
  location                  = "westeurope"
  resource_group_name       = "my-rg"
  app_service_plan_id       = "my-plan"
  storage_connection_string = "my-sa-connection-string"
  version                   = "~3"

  app_settings = {
    https_only                   = true
    FUNCTIONS_WORKER_RUNTIME     = "node"
    WEBSITE_NODE_DEFAULT_VERSION = "~12"
    # the secret sauce!
    # points to a zip file in a container
    WEBSITE_USE_ZIP              = "https://${azurerm_storage_account.mysa.name}.blob.core.windows.net/${azurerm_storage_container.my-ct.name}/my-code-1.0.1.zip${data.azurerm_storage_account_sas.my-sas.sas}"
  }
}

From the developer’s perspective, the Function release process looks like this:

Change code
Create a zip file with the source code, with a specific name like my-code-1.0.1.zip
Upload the zip to an Azure Container
Change the Terraform code to point to the uploaded file
Run terraform apply

Let’s assume the Function exposes a simple HTTP endpoint. If before step 4 we start sending traffic to the function – one request every half second – this is what we get:

This is my code v1.0.1! - 15:49:30
This is my code v1.0.1! - 15:49:31
This is my code v1.0.1! - 15:49:32
This is my code v1.0.1! - 15:49:32
This is my code v1.0.1! - 15:49:33
The service is unavailable. - 15:49:34
This is my code v1.0.2! - 15:49:54
This is my code v1.0.2! - 15:49:55
This is my code v1.0.2! - 15:49:56
This is my code v1.0.2! - 15:49:56
This is my code v1.0.2! - 15:49:57

During the terraform apply execution we have a ~20 seconds downtime! Not good!

Test #02 – Terraform + func ™️

Alright, the whole zip name changing dance should be causing downtime… func will solve this! That’s what I though.

In the second scenario, we still create the whole Functions App infrastructure with Terraform, but we omit the WEBSITE_USE_ZIP variable: instead, the developer will use the func CLI to deploy the Function itself, pointing to the Function App created by Terraform.

Again, the flow would look like this:

Deploy infrastructure with Terraform
Output the Function App name, let’s say fn-app
Within the Function repository, use the func CLI to deploy, like func azure functionapp publish fn-app
Update the source code
Deploy again with func

And, once again during the step 5, we got almost ~20 seconds of downtime :(

This is my code v1.0.1! - 15:59:20
This is my code v1.0.1! - 15:59:21
This is my code v1.0.1! - 15:59:22
This is my code v1.0.1! - 15:59:22
This is my code v1.0.1! - 15:59:23
The service is unavailable. - 15:59:24
The service is unavailable. - 15:59:25
This is my code v1.0.2! - 15:59:44
This is my code v1.0.2! - 15:59:45
This is my code v1.0.2! - 15:59:46
This is my code v1.0.2! - 15:59:46
This is my code v1.0.2! - 15:59:47

The standard way of developing and deploying Azure Functions – with the func CLI – will cause downtime D:

Although twenty seconds may OK for you, it’s a thing to be aware of.

Slots to the Rescue

There is one Azure way to solve it: Slots! Basically, it allows an user to deploy different Functions and hot swap them, without downtime. So one might have a Slot called prod with with a Function on the version 1.0.1 and another called staging with a Function named 1.0.2. Once staging is tested and ready to go, you swap them, so prod becomes 1.0.2.

Test #03 - Using slots

Create the two previous mentioned slots (you need a Resource Group with a Function App already created):
- az functionapp deployment slot create --name fn-app --resource-group rg --slot staging
- az functionapp deployment slot create --name fn-app --resource-group rg --slot prod
Zip your Function folder and deploy it to prod (you can only deploy Functions to Slots with a zip file and using the az CLI):
- az functionapp deployment source config-zip -g rg -n fn-app --src my-code-1.0.1.zip --slot prod
Change the source code and deploy it to the staging – like deploying a new version:
- az functionapp deployment source config-zip -g rg -n fn-app --src my-code-1.0.2.zip --slot staging
Once you are happy with the staging version, swap it with prod. Think about it like promoting the new version:
- az webapp deployment slot swap -g rg -n fn-app --slot staging --target-slot prod

If you continuously send requests to the prod Function during the swap, you’ll see something like this:

This is my code v1.0.1! - 17:10:22
This is my code v1.0.1! - 17:10:23
This is my code v1.0.1! - 17:10:23
This is my code v1.0.1! - 17:10:24
This is my code v1.0.2! - 17:10:24
This is my code v1.0.2! - 17:10:25
This is my code v1.0.1! - 17:10:25
This is my code v1.0.1! - 17:10:25
This is my code v1.0.1! - 17:10:25
This is my code v1.0.2! - 17:10:26
This is my code v1.0.2! - 17:10:27
This is my code v1.0.2! - 17:10:27
This is my code v1.0.1! - 17:10:28
This is my code v1.0.2! - 17:10:28
This is my code v1.0.2! - 17:10:29
This is my code v1.0.1! - 17:10:30
This is my code v1.0.1! - 17:10:31
This is my code v1.0.1! - 17:10:31
This is my code v1.0.2! - 17:10:32
This is my code v1.0.2! - 17:10:32
This is my code v1.0.2! - 17:10:33
This is my code v1.0.2! - 17:10:34
...

The Function is being hot swapped, and traffic is hitting both Functions until the swap is over. DONE! You’ve migrated your function without any downtime ✨

Let’s use Slots everywhere, right? Well, if you can afford, yes. As for now, Slots are only available for the Standard and Premium tiers. The only issue is the price: Functions in the Premium plan are charged by hour instead of execution. Check it out:

_Azure Functions Consumer Tier_

_Azure Functions Premium Tier_

So, yeah, as far as I know, Azure Functions with zero downtime is only for the ones who can afford.

@edit: Change to inform that Slots are only available in Standard and Premium tiers.

Tools of the Week #3

Mon, 11 Nov 2019 05:00:00 +0000

Tools of the week is a top of my head idea to write a tiny little bit about the tools and libraries I discovered every week. I love tools. I know you love them too ❤️

✨ This week tools are focused on Data Structures and Algorithms ✨

Data Structure Visualizations from USFCA

Platform to visualize how some algorithms and data structures work, including Stack, Queues, Sorting and more. It looks like a 2000s website, but it is very useful.

Visualgo

A more modern approach to live visualization of data structures and algorithms, where you can interact with the drawings, see how algorithms perform step-by-step while demonstrating with a code example and an e-Lecture Mode, where the platform adds explanations about the problem, how it works etc. VERY useful!

Big-O Cheat Sheet

An illustrated graph and list of the time complexity, space complexity, running time and all the Big-O related things from common Data Structures and Sorting Algorithms. Must have if you are learning the subject.

Pythontutor

Go through your code step-by-step and see what’s happening to your variables and functions in real-time. You can even go backwards. Besides the “Python” word in the name, you can use other languages like C, Java, Ruby and more.

A Telegram bot for CFP Land 🤖

Wed, 30 Oct 2019 05:00:00 +0000

Hello there!

A couple of months ago I made a Telegram bot for the amazing CFP Land platform 🤖 They list call for papers on conferences all over the world, send a weekly newsletter with new hot events, and also have a paid platform to organize your CFPs if you apply a lot! Pretty neat 🎉

Well, the bot is divided in two:

CFP Land Bot

An actual bot, which you can talk to. Kinda talk. It responds to some commands, like displaying all the CFP’s categories available or sending a list of the latest call for papers. Click here and say hi!

CFP Land Channel

A Telegram distribution channel, where I send a daily digest of their new CFPs. Just join here and wait for it!

As a quick note: the bot was running a couple of months ago but I totally over-engineer it over time. I got scared after adding lots of unnecessary infrastructure that I gave up :( BUT yesterday I decided to rip it off and give it a second chance.

Also, the project is open source 💃 You can check the code here. There are a lot of improvements, like docs to run locally and unit tests. Maybe one day :3

Python Idioms in Go

Mon, 21 Oct 2019 05:00:00 +0000

In the development world, Python is my native language. It all started in 2015 with a little project called itopy. Python is my favorite way to translate thought into applications due to its minimal learning curve, simplicity and elegance.

But like in real life, learning a second language has tons of benefits. You become able to navigate within other countries, talk to strangers across borders, understand their culture and way of living. It opens your mind to think differently about things and to construct new knowledge on top of established ideas. The last point is very important: naturally, we learn new concepts by comparing them to the ones we already know.

Studying a second programming language is no different. We compare their syntax, semantics, patterns and idioms. If something is the same, it just clicks! Otherwise, time is needed to build up knowledge.

In 2019 I choose Golang for my next adventure. It is vastly used in infrastructure projects (which I love!), has a community full of Pythonistas and is becoming very very popular. My path to get it was full of: “I can do X in Python, how do I do it in Go”? This article is a collection of those moments.

PS1: I won’t go in detail for the examples. A quick Googling will help you to understand them in depth.
PS2: I didn’t include import statements in Go for the sake of brevity.

Popping from a List

List are beautifully managed in Python:

words = ["guess", "whos", "back"]
word = words.pop()

print(word)
# back
print(words)
# ['guess', 'whos']

Slices – Go’s lists – are a bit more complicated, but the SliceTricks guide will help out.

words := []string{"guess", "whos", "back"}
word, words := words[len(words)-1], words[:len(words)-1]

fmt.Println(word)
// back
fmt.Println(words)
// [guess whos]

Pushing to a List

Pushing – or appending – is very similar:

🐍:

words = ["mangoes", "peaches", "limes"]
words.append("the sweet life")

print(words)
# ['mangoes', 'peaches', 'limes', 'the sweet life']

❗️🐍:

words := []string{"mangoes", "peaches", "limes"}
words = append(words, "the sweet life")

fmt.Println(words)
// [mangoes peaches limes the sweet life]

List Comprehension

Personally, the most beautiful sugary diet of Python:

numbers = [1,2,3,4]                        
double = [number * 2 for number in numbers]

print(double)
# [2, 4, 6, 8]

Unfortunately, in Go, more work is needed:

numbers := []int{1, 2, 3, 4}
var double []int

for _, number := range numbers {
    double = append(double, number*2)
}

fmt.Println(double)
// [2 4 6 8]

Dictionary Comprehension

Like a list comprehension, but iterating over values and keys in dictionary:

numbers = {'a': 1, 'b': 2, 'c': 3}
double = {key: value * 2 for key, value in numbers.items()}

print(double)
# {'a': 2, 'b': 4, 'c': 6}

Again, with Golang, more work:

numbers := map[string]int{"a": 1, "b": 2, "c": 3}
double := make(map[string]int)

for key, value := range numbers {
    double[key] = value * 2
}

fmt.Println(double)
// map[a:2 b:4 c:6]

Sets (and intersection)

Sets are like lists, but they have no duplicated items and some special properties. One of them is called intersection, which returns items showing up in both sets.

Python has a set data structure with all these features:

dna1 = {"loyalty", "royalty", "dna"}
dna2 = {"power", "poison", "pain", "dna"}

print(dna1 & dna2)
# {'dna'}

Sometimes Golang is too simple. It has no concept of set nor intersection. However, the same can be accomplish using a map and two loops. Not as elegant as Python, that’s for sure.

dna1 := make(map[string]bool)
dna1["loyalty"] = true
dna1["royalty"] = true
dna1["dna"] = true

dna2 := make(map[string]bool)
dna2["power"] = true
dna2["poision"] = true
dna2["pain"] = true
dna2["dna"] = true

intersection := make(map[string]bool)
for d1, _ := range dna1 {
    _, exists := dna2[d1]
    if exists {
        intersection[d1] = true
    }
}

fmt.Println(intersection)
// map[dna:true]

Thanks Tamás for the tip on using a key check on dna2!

Enumerating

In Python, we use enumerate() to loop over a list with its index:

words = ["i'll", "tell", "what", "I", "want"]

for index, word in enumerate(words):
    print(index, word)

In Go, this is the default behavior:

words := []string{"i'll", "tell", "what", "I", "want"}

for index, word := range words {
    fmt.Println(index, word)
}

Output:

i'll
tell
what
I
want

x in y

Python’s in operator is used to tell if something is inside another thing, like in a list:

words = ["i'm", "a", "bad", "guy"]
if "bad" in words:
    print("duh!")

# duh!

Or to verify if a key exists in a dict:

words = {"tough": "guy", "rough": "guy", "enough": "guy"}
if "tough" in words:
    print("billie is the best")

# billie is the best

Go has no in operator 😢 The verbose road needs to be taken:

words := []string{"i'm", "a", "bad", "guy"}

for _, word := range words {
    if word == "bad" {
        fmt.Println("duh!")
    }
}

// duh!

A bit of sugar is available to check keys in a map:

words := map[string]string{"tough": "guy", "rough": "guy", "enough": "guy"}

_, ok := words["tough"]
if ok {
    fmt.Println("billie is the best")
}

// billie is the best

Lambda Functions

Lambdas, or anonymous functions, are nameless functions usually doing only one very simple thing. In Python, a lambda function any number of arguments but is restricted to a single expression – just one thing can happen after the ::

multiply = lambda x: x*x

print(multiply(2))
# 4

Golang also has lambda functions, but they can spawn multiple statements:

double := func(x int) int { return x * 2 }

fmt.Println(double(2))
// 4

printAndDouble := func(x int) int { fmt.Println("one thing"); return x * 2}

fmt.Println(printAndDouble(2))
// one thing
// 4

Context Manager

A context manager is a very powerful pattern in Python. It’s usually used to manage resources, like closing a file automatically so you don’t forget it. This is a very simplified explanation, it can do a lot more. Here is a link for further explanation.

with open('/etc/passwd', 'r') as file:
    print(file.read())

# all your passwords 👺

file.read()
# ValueError: I/O operation on closed file.

Golang has a kinda similar concept called defer. It ensures that a function call is performed at the end of the current function, so you don’t forget to close your file after you’ve opened it.

func main() {
    file, _ := os.Open("/etc/passwd")
    // called when this function block finishes 
    defer file.Close()

    data, _ := ioutil.ReadAll(file)

    fmt.Println(data)
    // all your passwords, in bytes 👺
}

Assignment Expressions

In Python 3.8, assignment expressions define a new notation: NAME := expr, where you can assign a value to a variable during an expression. An example is using the if statement:

def x():
    return 10

if (number := x()) is number == 10:
    print(10)

# 10

Go also supports it:

func x() int { return 10 }

func main() {
    if x := x(); x == 10 {
        fmt.Println(x)
    }
}

// 10

Classes with initialization and methods

Python has Classes to create a new type with properties and methods, which can be used to create instances of that type:

class Double: 
    def __init__(self, x): 
        self.x = x 
 
    def double(self): 
        return self.x * 2 
 
instance = Double(2) 
 
print(instance.double())     
# 4

You can accomplish basically the same with Golang’s struct:

type Double struct {
    x int
}

func (d *Double) double() int {
    return d.x * 2
}

func main() {
    instance := Double{x: 2}

    fmt.Println(instance.double())
    // 4
}

Sugary Python, Simple Go

Even though I don’t add sugar to my coffee, I’d pour some over Go. Learning Go after Python was full of surprises: I was used to a lot of tricks and one-liners. To this day, I read the SliceTricks page more than I’d be proud of.

Besides frustrations here and there, I love Go. Statically typed, easy coroutines, single binary, multi-architecture compilation and more! It has so many great features!

Both languages have their place in my tool belt and my heart ❤️

Thanks to the people from Twitter and DEV to share their favorite idioms 💙

Share your favorite idioms in the comments below and let’s try to find its Go-like implementation ✨

Implementing Dijkstra Algorithm in Go

Wed, 16 Oct 2019 05:00:00 +0000

I didn’t go through Computer Science, had a very early dropout. Since I started working in the tech space, I cared a lot about RESTful APIs, preventing systems to fail. Algorithms and data structures always seemed daunting and terrifying.

After picking up the Grokking Algorithms book and giving up three times, I finally decided to put some real effort into it. And it’s being an amazing adventure, I can’t recommend it enough!

Chapter 7 is all about the famous Dijkstra Algorithm: finding the shortest path for a vertice in a graph. A bunch of pomodoros later, I got it. The code below is as commented as possible, to solidify the knowledge in my head and help others as well. I also wrote unit tests and the answers to the book’s exercises here.

I won’t go in much detail explaining the theory behind it, other resources do a better job I’d be capable of. Start with this quick video, take a look at Brilliant’s article and, of course, read the already mentioned book if you can.

It probably can be improved, as everything can. I think this is a nice balance between “working” and “readable”. If you have any nice tips to contribute, please leave it in the comments ✨

Well, here it is, the Dijkstra Algorithm in Go:

package main

import (
	"fmt"
	"sort"
	"strconv"
)

type Graph struct {
	Edges []*Edge
	Nodes []*Node
}

type Edge struct {
	Parent *Node
	Child  *Node
	Cost   int
}

type Node struct {
	Name string
}

const Infinity = int(^uint(0) >> 1)

// AddEdge adds an Edge to the Graph
func (g *Graph) AddEdge(parent, child *Node, cost int) {
	edge := &Edge{
		Parent: parent,
		Child:  child,
		Cost:   cost,
	}

	g.Edges = append(g.Edges, edge)
	g.AddNode(parent)
	g.AddNode(child)
}

// AddNode adds a Node to the Graph list of Nodes, if the the node wasn't already added
func (g *Graph) AddNode(node *Node) {
	var isPresent bool
	for _, n := range g.Nodes {
		if n == node {
			isPresent = true
		}
	}

	if !isPresent {
		g.Nodes = append(g.Nodes, node)
	}
}

// String returns a string representation of the Graph
func (g *Graph) String() string {
	var s string

	s += "Edges:\n"
	for _, edge := range g.Edges {
		s += edge.Parent.Name + " -> " + edge.Child.Name + " = " + strconv.Itoa(edge.Cost)
		s += "\n"
	}
	s += "\n"

	s += "Nodes: "
	for i, node := range g.Nodes {
		if i == len(g.Nodes)-1 {
			s += node.Name
		} else {
			s += node.Name + ", "
		}
	}
	s += "\n"

	return s
}

// Dijkstra implements THE Dijkstra algorithm
// Returns the shortest path from startNode to all the other Nodes
func (g *Graph) Dijkstra(startNode *Node) (shortestPathTable string) {

	// First, we instantiate a "Cost Table", it will hold the information:
	// "From startNode, what's is the cost to all the other Nodes?"
	// When initialized, It looks like this:
	// NODE  COST
	//  A     0    // The startNode has always the lowest cost to itself, in this case, 0
	//  B    Inf   // the distance to all the other Nodes are unknown, so we mark as Infinity
	//  C    Inf
	// ...
	costTable := g.NewCostTable(startNode)

	// An empty list of "visited" Nodes. Everytime the algorithm runs on a Node, we add it here
	var visited []*Node

	// A loop to visit all Nodes
	for len(visited) != len(g.Nodes) {

		// Get closest non visited Node (lower cost) from the costTable
		node := getClosestNonVisitedNode(costTable, visited)

		// Mark Node as visited
		visited = append(visited, node)

		// Get Node's Edges (its neighbors)
		nodeEdges := g.GetNodeEdges(node)

		for _, edge := range nodeEdges {

			// The distance to that neighbor, let's say B is the cost from the costTable + the cost to get there (Edge cost)
			// In the first run, the costTable says it's "Infinity"
			// Plus the actual cost, let's say "5"
			// The distance becomes "5"
			distanceToNeighbor := costTable[node] + edge.Cost

			// If the distance above is lesser than the distance currently in the costTable for that neighbor
			if distanceToNeighbor < costTable[edge.Child] {

				// Update the costTable for that neighbor
				costTable[edge.Child] = distanceToNeighbor
			}
		}
	}

	// Make the costTable nice to read :)
	for node, cost := range costTable {
		shortestPathTable += fmt.Sprintf("Distance from %s to %s = %d\n", startNode.Name, node.Name, cost)
	}

	return shortestPathTable
}

// NewCostTable returns an initialized cost table for the Dijkstra algorithm work with
// by default, the lowest cost is assigned to the startNode – so the algorithm starts from there
// all the other Nodes in the Graph receives the Infinity value
func (g *Graph) NewCostTable(startNode *Node) map[*Node]int {
	costTable := make(map[*Node]int)
	costTable[startNode] = 0

	for _, node := range g.Nodes {
		if node != startNode {
			costTable[node] = Infinity
		}
	}

	return costTable
}

// GetNodeEdges returns all the Edges that start with the specified Node
// In other terms, returns all the Edges connecting to the Node's neighbors
func (g *Graph) GetNodeEdges(node *Node) (edges []*Edge) {
	for _, edge := range g.Edges {
		if edge.Parent == node {
			edges = append(edges, edge)
		}
	}

	return edges
}

// getClosestNonVisitedNode returns the closest Node (with the lower cost) from the costTable
// **if the node hasn't been visited yet**
func getClosestNonVisitedNode(costTable map[*Node]int, visited []*Node) *Node {
	type CostTableToSort struct {
		Node *Node
		Cost int
	}
	var sorted []CostTableToSort

	// Verify if the Node has been visited already
	for node, cost := range costTable {
		var isVisited bool
		for _, visitedNode := range visited {
			if node == visitedNode {
				isVisited = true
			}
		}
		// If not, add them to the sorted slice
		if !isVisited {
			sorted = append(sorted, CostTableToSort{node, cost})
		}
	}

	// We need the Node with the lower cost from the costTable
	// So it's important to sort it
	// Here I'm using an anonymous struct to make it easier to sort a map
	sort.Slice(sorted, func(i, j int) bool {
		return sorted[i].Cost < sorted[j].Cost
	})

	return sorted[0].Node
}

func main() {
	a := &Node{Name: "a"}
	b := &Node{Name: "b"}
	c := &Node{Name: "c"}
	d := &Node{Name: "d"}
	e := &Node{Name: "e"}
	f := &Node{Name: "f"}
	g := &Node{Name: "g"}

	graph := Graph{}
	graph.AddEdge(a, c, 2)
	graph.AddEdge(a, b, 5)
	graph.AddEdge(c, b, 1)
	graph.AddEdge(c, d, 9)
	graph.AddEdge(b, d, 4)
	graph.AddEdge(d, e, 2)
	graph.AddEdge(d, g, 30)
	graph.AddEdge(d, f, 10)
	graph.AddEdge(f, g, 1)

	fmt.Println(graph.Dijkstra(a))
}

The output will be:

Distance from a to a = 0
Distance from a to c = 2
Distance from a to b = 3
Distance from a to d = 7
Distance from a to e = 9
Distance from a to g = 18
Distance from a to f = 17

I’m so happy 😌

@edit: The code above is the version 1. Due community contributions (thanks!), it improved! Get the latest version here.

Tools of the Week #2

Mon, 14 Oct 2019 05:00:00 +0000

Tools of the week is a top of my head idea to write a tiny little bit about the tools and libraries I discovered every week. I love tools. I now you love them too ❤️

sls-dev-tools

Like htop, but for AWS Lambda functions. Honestly, I had this idea a while back! I’m thrilled to find out that other people made it ✨

Within the console app a lot of information is available, like a list of functions, metrics, logs, invocation duration, location in a map and more!

🚨 PRICE ALERT 🚨 The tool calls AWS API, a lot, especially GetMetricData from CloudWatch, costing $0.01 per 1k metrics requested.

kubeinvaders

THE👏BEST👏CHAOS👏ENGINEERING👏TOOL👏EVER👏

IT’S SPACE INVADERS

BUT THE ALIENS

ARE PODS

I’M TOO EXCITED TO TURN OFF THE CAPS LOCK!!!

ENJOY IT.

stern

This week do yourself a favor and forget kubectl logs, really, pretend it never existed.

Stern is the best tool I know of for tailing logs in K8S, working with multiple Pods AND multiple containers. Its filtering system understands Pods without their “unique Deployment ID”, so $ stern prometheus-server just works.

inlets

Inlets is like ngrok, but open source (and better). Basically, you spin up a VM somewhere with an external IP address and run inlets in server mode. On your local machine, execute inlets specifying the VM’s IP address and your local port. BOOM! Tunneling!

🍀 BONUS: Check inlets-operator. It does the same tunneling dance, but you end up with a public LoadBalancer for your local Kubernetes cluster. A M A Z I N G.

reverse-interview

It’s time to flip the table and learn about questions you should make to your interviewer, including subjects like the company’s role, tech stack, potential coworkers and more, nicely translated into different languages ⚡️