How to fully automate backend infrastructure generation

October 29, 2022 · 33 min read

Data Scientist

Cover image

If you're tired of manually creating AWS resources and spending hours on infrastructure tasks, here is a walkthrough of how you can automate all the tedious work, so you can spend your time on more important tasks.

I've built and launched 5 different micro-SaaS on Product Hunt, and one of the most tedious parts is generating the infrastructure and generating the backend. I'm deploying everything on AWS, so I always have the same set of resources I'm using:

AWS resources

I used to create each resource manually in the AWS dashboard, which took me about ~2 hours. Then I finally switched to infrastructure as code, using YML files with AWS Cloudformation (similar to Terraform but AWS native).

YML files and Cloudformation saved me a whole hour, but it still took me ~1 hour from start to finish.

The biggest problem is not really the hour it takes to add all the resources and deploy. My biggest concern was mainly copying and pasting different resource snippets from different files and always getting a variable wrong and debugging.

So I finally sat down and

1. built a simple frontend for picking all the resources and

2. wrote a script that generates all the code needed.

Some of the most tedious tasks are now fully automated:

- Generating YAML files for all the resources - Creating infrastructure files - Creating repos (GitHub API) - Creating branches (GitHub API) - Creating CI/CD pipelines (GitHub actions) - Adding encrypted secrets (GitHub API) - Creating pull requests (GitHub API) - Merging pull requests (GitHub API)

I went from ~2 hours to ~10 minutes:

AWS resource generation

I shared this process during two hackathons:

Hackathon tweet

With these replies:

Hackathon tweet replies

Here are the steps we'll go through in this guide and fully automate with Python scripts:

Now that we have the background let's look at how I built the system.

This walkthrough is divided into 3 parts:

1. Resources needed

2. Building a frontend to pick resources

- ⚛️ ReactJS

3. Scripts to generate backend

- 🐍 Python

1. Resources needed

When I build a new SaaS, I either go with just a landing page with a demo + authorization or a complete app with authentication and authorization (login/signup). Here are the resources depending on the approach:

Type	Resource	Landing page with demo	Web app
Authorization	Cognito IdentityPool	✅	✅
Authentication	Cognito UserPool	❌	✅
Storage	S3 bucket landing page	✅	✅
Storage	S3 bucket landing page www	✅	✅
Storage	S3 bucket app	❌	✅
Storage	S3 bucket app www	❌	✅
Storage	S3 bucket public files	✅	✅
Storage	S3 bucket deployment	✅	✅
Storage	S3 bucket user uploads	❌	✅
CDN	Cloudfront landing page	✅	✅
CDN	Cloudfront landing page www	✅	✅
CDN	Cloudfront app	❌	✅
CDN	Cloudfront app www	❌	✅
CDN	Cloudfront public files	✅	✅
Database	DynamoDB	❌	✅
Total		8	15

Below is a quick summary of the resources. If you're already familiar with all of them, you can skip to the part where I'm building the frontend.

Authorization | Cognito

Amazon Cognito lets you easily add user sign-up and authentication to your mobile and web apps. Amazon Cognito also enables you to authenticate users through an external identity provider and provides temporary security credentials to access your app's backend resources in AWS or any service behind Amazon API Gateway.

When building landing pages with backend access, I use Cognito federated identities to authorize unauthenticated users' access to the backend. When making web apps with a login/signup flow, I use user pools to enable users to access their backend data and other backend resources securely.

Storage | S3 buckets

An Amazon S3 bucket is a public cloud storage resource available in Amazon Web Services' (AWS) Simple Storage Service (S3), an object storage offering. Amazon S3 buckets, which are similar to file folders, store objects, which consist of data and its descriptive metadata.

I'm using multiple S3 buckets to store for static website hosting, one for the landing page and one for the web app, and one for redirecting from www. Except for static website hosting, I'm also using buckets for public files, like social media cards and logos.

CDN | Cloudfront

Amazon CloudFront is a content delivery network. Amazon CloudFront is a web service that speeds up distribution of your static and dynamic web content, such as .html, .css, .js, and image files, to your users. CloudFront delivers your content through a worldwide network of data centers called edge locations. When a user requests content that you're serving with CloudFront, the request is routed to the edge location that provides the lowest latency (time delay), so that content is delivered with the best possible performance.

Cloudfront is a CDN for static website hosting, but I'm also using Cloudfront for public files and images.

Database | DynamoDB

Amazon DynamoDB is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability. DynamoDB lets you offload the administrative burdens of operating and scaling a distributed database so that you don't have to worry about hardware provisioning, setup and configuration, replication, software patching, or cluster scaling.

NoSQL is a hill I'm willing to die on.

Alrighty, we have all the resources needed for a first starting point, let's look at the system I built to generate the YML files for Cloudformation.

2. Building a frontend to pick resources

Create the Cloudformation YML

The step I took to reduce my time copying and pasting resources into a YML file was actually to build a tiny web app where I pick each resource I need.

Here's an overview of the web app:

Web app overview

You have 5 tabs for parameters, S3 (storage), Cloudfront (CDN), Cognito (authentication/authorization) and DynamoDB (database/table):

Web app tabs

You start by filling out the parameters for your app; these will be used as variables in the final infrastructure as code Cloudformation YML file:

Web app parameters

They'll be at the top of the YML file you'll get when you click to download the files:

Parameters:
  App:
    Description: The app name
    Default: app
    Type: String

  DomainName:
    Description: The domain name
    Default: myprojectname.com
    Type: String

  DomainNameWWW:
    Description: The www domain name
    Default: www.myprojectname.com
    Type: String

  DomainNameApp:
    Description: The app domain name
    Default: app.myprojectname.com
    Type: String

  DomainNameAppWWW:
    Description: The www app domain name
    Default: www.app.myprojectname.com
    Type: String

  Id:
    Description: The id for uniqueness
    Default: 44pz4
    Type: String

  LandingPage:
    Description: The landing page
    Default: landing-page
    Type: String

  ProjectName:
    Description: The project name
    Default: my project name
    Type: String

  ProjectShortName:
    Description: The project short name
    Default: pn
    Type: String

The first tab is for S3, the storage; here you pick the static website hosting and if you need a bucket for public files. You can see a list of your selected resources to the right:

Web app buckets

The second tab is for CloudFront (CDN), here you pick if you need CDN for both the web app or just the landing page:

Web app cloudfront

The third tab is for Cognito (authentication/authorization). You can choose federated identity for landing pages with access to the backend or a user pool for web apps. I made identity pool included for user pool:

Web app cognito

The fourth and last tab is for NoSQL DynamoDB (database/table):

Web app dynamodb

I usually only have a table for my web apps. Still you could easily add an additional table for the landing page if you have an interactive demo for unauthenticated users and want to save their input data:

Web app dynamodb multiple

If you go ahead and click on Download cloudformation yml, you'll receive an entire YML file for your infrastructure as code, including all policies for each resource looking something like this:

Parameters:
  App:
    Description: The app name
    Default: app
    Type: String

  DomainName:
    Description: The domain name
    Default: myprojectname.com
    Type: String

#################################
####### 9 more parameters #######
#################################

Resources:

# Landing page analytics bucket
  AnalyticsBucket:
    Condition: DevCondition
    Type: AWS::S3::Bucket
    Properties:
      AccessControl: Private
      BucketName: !Sub \${ProjectName}-analytics-\${Id}
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true

#################################
####### 10 more resources #######
#################################

You'll also download a JSON file with metadata for when we'll run the automated scripts later in this guide:

{
    "aws_profile": "aws_profile",
    "app_description": "My project name",
    "domain_name": "myprojectname.com",
    "project_name": "my project name",
    "project_name_short": "pn",
    "resource_id": "wnz13"
}

Alright, so far, we've generated the infrastructure as code YML needed to deploy all your backend resources. The next step is to automate the creation of GitHub repos.

3. Python scripts

In this part of the guide, we'll create all the scripts needed to deploy and generate the backend.

1. Script for defining project variables

Now that we have the YML file, the next step is constructing the Python scripts needed to deploy the infrastructure.

Start by creating all these file and folder structure to follow along the guide:

Python script folder structure

We'll primarily work with main.py, the main Python script that will orchestrate the creation of all other files and GitHub repos.

Starting with the main function in main.py add the following starting point:

import argparse
def generate_mono_repo(args):
 """
 Generate a mono repo with resources in AWS deployed to GitHub
 """

 try:
    print("args: ", args)

 except Exception as e:
    print("fail: ", e)

if __name__ == '__main__':
    parser = argparse.ArgumentParser()

    parser.add_argument('--yml_path', help='The template yml path')
    parser.add_argument('--json_path', help='The template json path')

    args = parser.parse_args()

    generate_mono_repo(args)

We'll have two arguments; the path to the YML file and the path to the JSON file we generated and downloaded in the last step.

Add the following function to blueprint.py:

import json
import os

from pathlib import Path


def get_project_variables(json_path, yml_path):
    """
    Create all the variables needed for generating a new project
    :return:
    """

    # Read pre-created JSON
    parent_path = str(Path().resolve().parent)

    json_file_path = os.path.join(parent_path, 'infrastructure-automation-guide',
                                  json_path)

    with open(json_file_path) as f:
        project_data = json.load(f)

    repo_folder_name = f"{project_data['project_name_short']}"

    # Create project variables
    project_variables = {
        "cloudformation_data": os.path.join(str(Path().resolve().parent),
                                            'infrastructure-automation-guide',
                                            yml_path),
        "dev_stage": 'dev',
        "file_name_cloudformation": 'cloudformation.yml',
        'infrastructure_folder_path': os.path.join(parent_path,
                                                   repo_folder_name, f"{project_data['project_name_short']}-infrastructure"),
        "prod_stage": 'prod',
        "parent_path": parent_path,
        "region": 'us-east-1',
        "repo_folder_name": repo_folder_name,
        "repo_folder_path": os.path.join(parent_path,
                                         repo_folder_name),
        "workflows_folder_path": os.path.join(
            parent_path,
            repo_folder_name,
            '.github',
            'workflows')
    }

    # Add the json data to the project variables
    project_variables.update(project_data)

    return project_variables

This function will create the needed project variables, now call that function from main.py:

# Add to main.py
project_variables = blueprint.get_project_variables(
    json_path=args.json_path, 
    yml_path=args.yml_path
)

2. Script for mono-repo folder structure creation

The next step is to write a script to create the folder structure for the mono-repo that you'll deploy to GitHub later. After calling the following function, you'll have this folder structure created:

Mono repo folder structure

In the file file_creation.py, add the following:

import os

from pathlib import Path


def create_folder_structure_mono_repo(project_variables):
    """
    Generate the folder structure for a mono-repo
    :return:
    """

    # Create path to infrastructure folder
    infrastructure_folder = os.path.join(
        project_variables['repo_folder_path'],
        f"{project_variables['project_name_short']}-infrastructure")

    # Create path to backend folder
    backend_folder = os.path.join(
        project_variables['repo_folder_path'],
        f"{project_variables['project_name_short']}-backend")

    # Create path to frontend folder
    frontend_folder = os.path.join(
        project_variables['repo_folder_path'],
        f"{project_variables['project_name_short']}-frontend")

    # Create the folders
    Path(project_variables['repo_folder_path']).mkdir(parents=True, exist_ok=True)
    Path(infrastructure_folder).mkdir(parents=True, exist_ok=True)
    Path(backend_folder).mkdir(parents=True, exist_ok=True)
    Path(frontend_folder).mkdir(parents=True, exist_ok=True)

Then call this function from the main.py:

# Add to main.py
file_creation.create_folder_structure_mono_repo(
    project_variables=project_variables
)

Visit your parent folder, you should now have 3 new folders in a folder named after the short name of your project, as shown before:

Mono repo folder structure

We can now check the second checkbox:

3. Script for creating folder structure for CI/CD in GitHub Actions

The next step is to create a script that will make the folders needed for CI/CD with GitHub action. You can find the documentation here https://docs.github.com/en/actions.

The folder structure we'll create with our script will be like this when we're finished:

GitHub Action workflow folder structure

Each yml-file will have its specific GitHub actions workflow.

Let's start with creating the .github folder and the workflows folder by adding the following function to file_creation.py:

def create_workflows_folder_structure_mono_repo(project_variables):
    """
    Generate the workflows folder structure for a mono-repo
    :return:
    """

    # Create the workflows folders from the path
    Path(project_variables['workflows_folder_path']).mkdir(parents=True, exist_ok=True)

Add a function call to create_workflows_folder_structure_mono_repo in main.py:

# Add to main.py
file_creation.create_workflows_folder_structure_mono_repo(
    project_variables=project_variables
)

If you look in your directory, you should see the newly created folder structure for GitHub actions CI/CD:

GitHub Action workflow folder

4. Script for creating the CI/CI workflow for infrastructure

Now let's create the infrastructure.yml file, which will contain the instructions for your CI/CD workflow. Add the following function in gitbub_actions.py:

from ruamel.yaml import YAML

def generate_workflow_infrastructure_mono_repo(project_variables):
    """
    Generate the workflow for the infrastructure github actions
    :return:
    """

    workflow = f"""\
    name: {project_variables["project_name_short"]}-infrastructure
    on:
      push:
        paths:
        branches: 
      pull_request:
        paths:
        branches:

    env:
      AWS_ACCESS_KEY_ID: \${{{{ secrets.AWS_ACCESS_KEY_ID }}}}
      AWS_SECRET_ACCESS_KEY: \${{{{ secrets.AWS_SECRET_ACCESS_KEY }}}}
      AWS_DEFAULT_REGION: us-east-1
      AWS_REGION: us-east-1

    jobs:
      build:
        runs-on: ubuntu-latest


        steps:

    """

    yaml = YAML()
    workflow_yml = yaml.load(workflow)

    # Add variables
    workflow_yml['on']['push']['paths'] = [f'{project_variables["project_name_short"]}-infrastructure/**',
                                           '.github/workflows/infrastructure.yml']
    workflow_yml['on']['push']['branches'] = ['*']

    workflow_yml['on']['pull_request']['paths'] = [f'{project_variables["project_name_short"]}-infrastructure/**',
                                           '.github/workflows/infrastructure.yml']
    workflow_yml['on']['pull_request']['branches'] = ['*']
    workflow_yml['jobs']['build']['steps'] = [{
        "uses": "actions/checkout@v2"
    },
        {
            "name": "Run cloudformation develop",
            "working-directory": f'./{project_variables["project_name_short"]}-infrastructure',
            "if": "github.ref == 'refs/heads/develop'",
            "run": f'aws cloudformation deploy --stack-name {project_variables["project_name_short"]}-infrastructure-{project_variables["dev_stage"]} --template-file {project_variables["file_name_cloudformation"]} --parameter-overrides EnvType={project_variables["dev_stage"]} --region {project_variables["region"]} --capabilities CAPABILITY_IAM --no-fail-on-empty-changeset'
        },
        {
            "name": "Run cloudformation production",
            "working-directory": f'./{project_variables["project_name_short"]}-infrastructure',
            "if": "github.ref == 'refs/heads/main'",
            "run": f'aws cloudformation deploy --stack-name {project_variables["project_name_short"]}-infrastructure-{project_variables["prod_stage"]} --template-file {project_variables["file_name_cloudformation"]} --parameter-overrides EnvType={project_variables["prod_stage"]} --region {project_variables["region"]} --capabilities CAPABILITY_IAM --no-fail-on-empty-changeset'
        }]

    return workflow_yml

This function will create the data needed for deploying the Cloudformation infrastructure as code to AWS that we generated in the very first step.

Call the function from the main.py file to create the yml data:

# Add to main.py
workflow_infrastructure_yml_file = github_actions.generate_workflow_infrastructure_mono_repo(
            project_variables=project_variables
)

Now that we have the data for the workflow, we need a function that can save it to file locally in your repo. Create the following helper function in your file_creation.py file:

import sys

from ruamel.yaml import YAML

def save_yaml_file_locally(file_path, yml_file):
    """
    Save a file locally
    :return:
    """

    # Save the cloudformation file
    YAML().dump(yml_file, sys.stdout)
    with open(file_path, 'w') as file:
        document = YAML().dump(yml_file, file)

Finally, call the function in main.py:

# Add to main.py
file_creation.save_yaml_file_locally(
            file_path=os.path.join(project_variables['workflows_folder_path'], 'infrastructure.yml'),
            yml_file=workflow_infrastructure_yml_data)

If you look in your parent folder where you have your project after you run this function, you should now see the infrastructure.yml file created:

Infrastructure workflow yml

Great, we're done with the CI/CD pipeline:

5. Script for creating a gitignore file

The next step is to create a script that will generate a git ignore file for the GitHub repo. Add the following code to your file_creation.py file:

def create_gitignore():
    """
    Create a gitignore file for the infrastructure
    :return:
    """

    gitignore = """
# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.

# dependencies
node_modules
/node_modules
/.pnp
.pnp.js

# testing
coverage
/coverage

# production
/build
build
dist

# misc
.DS_Store
.env
.env.local
.env.development.local
.env.test.local
.env.production.local
**/env.yml
*/env.yml
/env.yml
*.pem
.vscode/

npm-debug.log*
yarn-debug.log*
yarn-error.log*

#Python
.idea/
.requirements.zip
.serverless/
__pycache__/

# Env
env.yml

# Stripe
/stripe

.requirements/
    """

    return gitignore

Also, add a new helper function for saving files locally to file_creation.py:

def save_file_locally(file_data, file_path):
    """
    Save a file locally with format in name
    :return:
    """

    with open(file_path, 'w') as file:
        file.write(file_data)

Finally, call the functions from main.py to create a .gitignore and to save the file locally:

# Add to main.py

# Create git ignore file
git_ignore = file_creation.create_gitignore()

# Save git ignore locally
file_creation.save_file_locally(
    file_data=git_ignore,
    file_path=os.path.join(
        project_variables['repo_folder_path'],
        '.gitignore'
        )
    )

You should now have a .gitignore in your parent folder:

Gitignore

6. Script for creating README file

Let's add the script for creating a new README file for your GitHub repo. Add the following snippet to file_creation.py:

def create_read_me_file(file_path, paragraph, title):
    """
    Create a read me file
    :return:
    """

    mdFile = MdUtils(file_name=file_path)
    mdFile.new_header(level=1, title=f'{title}')
    mdFile.new_paragraph(f"{paragraph}")
    mdFile.create_md_file()

Call the new function from your main.py file:

# Add to main.py
file_creation.create_read_me_file(
    file_path=os.path.join(
        project_variables['repo_folder_path'], 
        'README.md'),
    paragraph=f"The mono-repo of {project_variables['project_name']}",
    title=f"{project_variables['project_name']} infrastructure"
)

Make sure you now have a REAME.md file in your file system:

README

7. Script for creating CloudFormation yml

We have one last function to create before the repo is ready to be pushed to your GitHub repo and deployed to AWS. Start by creating a function that reads the YML infrastructure file we created in the first step. Add the following to infrastructure.py:

from ruamel.yaml import YAML

def read_infrastructure_yml(project_variables):
    """
    Read the infrastructure YML file
    :return:
    """

    # Read the input infrastructure file
    yaml = YAML()
    with open(project_variables['cloudformation_data']) as file:
        yml_file = yaml.load(file)

    return yml_file

Call the function from the main.py file and save the data locally:

# Add to main.py
file_creation.save_yaml_file_locally(
    file_path=os.path.join(project_variables['infrastructure_folder_path'],
                           project_variables['file_name_cloudformation']),
    yml_file=yml_data)

You should now see the newly created cloudformation.yml in your infrastructure folder:

cloudformation yml

8. Script for creating a new GitHub repo

For this section, you'll need your GitHub username and a token; read this guide to get a token: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token

Once you have your token, add your GitHub username and the token to the project_variables dict in the blueprint.py file:

# Add to project_variables in blueprint.py
"owner": YOUR_GITHUB_USERNAME,
"token": YOUR_GITHUB_TOKEN,
"repo_name": project_data['project_name'].lower().replace(' ', '-'),

Continue by adding this function to github.py:

import json
import requests

def create_new_repo_from_scratch(repo_name, token):
    """
    Create a new repo from scratch
    """
    query_url = "https://api.github.com/user/repos"
    headers = {
        'Accept': 'application/vnd.github.v3+json"', 
        'Authorization': f'token {token}'
        }
    params = {
        "name": f"{repo_name}", 
        "private": True
        }
    r = requests.post(query_url, headers=headers, data=json.dumps(params))

    return r.json()

This function creates a new repo on your GitHub account. Make sure the repo is private by using the parameter "private": True, as seen above.

Add a call to this function in main.py:

# Add to main.py
response = github.create_new_repo_from_scratch(
    repo_name=project_variables['repo_name'],
    token=project_variables['token']
)

Check the GitHub response to make sure you have the right combination of username and GitHub token by adding the following snippet to main.py:

# Add to main.py
import sys

if 'message' in response:
    sys.stdout.write("GitHub infrastructure creation response: \n\n" + response['message'])
else:
    sys.stdout.write("GitHub infrastructure creation is finished: \n\n")

Go ahead and visit your GitHub repos at github.com. You should see a newly created repo called your-project-name.

We're done with the script that is creating new repos:

9. Script for initiating and pushing to the repo

Let's initiate it and all the files now that you have a new repo.

Add the following function to github.py:

def git_init_push_new_repo(commit_comment, owner, repo_name, repo_path):
    """
    Git init a new repo, add a main branch, add a README, make the first commit and push to main
    """

    # Initialize git
    os.system(f"cd {repo_path} && git init")
    os.system(f"cd {repo_path} && git add README.md")
    os.system(f"cd {repo_path} && git commit -m '{commit_comment}'")

    # Rename main branch to main
    os.system(f"cd {repo_path} && git branch -M main")

    # Remote add and push
    os.system(f"cd {repo_path} && git remote add origin https://github.com/{owner}/{repo_name}.git")
    os.system(f"cd {repo_path} && git push -u origin main")

This script will initiate the repo, add your README file and commit it. Then it will rename the main branch to main. Lastly, it will add the origin and push to the main.

Call this function from your main.py file:

# Add to main.py
github.git_init_push_new_repo(commit_comment="Added README",
                              owner=project_variables['owner'],
                              repo_name=project_variables['repo_name'],
                              repo_path=project_variables['repo_folder_path'])

If you visit your GitHub repo, you'll see the README file pushed to the main branch.

Check the box for pushing to origin:

10. Script for fetching the public key of a repo

Later on in this guide, we'll want to push encrypted repo secrets. Secrets like your access keys to AWS to be able to deploy your backend resources.

For this, we'll need to get the public key of the newly created repo.

Add this helper function to github.py:

def fetch_public_key_repo(owner, repo_name, token):
    query_url = f"https://api.github.com/repos/{owner}/{repo_name}/actions/secrets/public-key"
    headers = {
        'Accept': 'application/vnd.github.v3+json"', 
        'Authorization': f'token {token}'
        }
    params = {
        'owner': owner,
        'repo': repo_name
    }
    public_key = requests.get(
        query_url, 
        headers=headers, 
        data=json.dumps(params)
    )
    return json.loads(public_key.text)

Call the function from the main.py file:

# Add to main.py
github_repo_key = github.fetch_public_key_repo(
    owner=project_variables['owner'],
    repo_name=project_variables['repo_name'],
    token=project_variables['token']
)

11. Script for generating encrypted AWS secrets

To successfully deploy and create your infrastructure resources in AWS, you'll need to add your AWS credentials to your GitHub repo. In this step, we'll generate encrypted secrets that we'll add to the repo.

Add the following function to github.py:

from base64 import b64encode
from nacl import encoding, public

def generate_encrypted_secret(github_public_key, secret_to_encrypt):
    public_key = public.PublicKey(
        github_public_key.encode("utf-8"), 
        encoding.Base64Encoder()
    )
    sealed_box = public.SealedBox(public_key)

    secret = sealed_box.encrypt(secret_to_encrypt.encode("utf-8"))
    secret_encrypted = b64encode(secret).decode("utf-8")

    return secret_encrypted

Call this function and create encrypted secrets for your AWS access key and AWS secret key in your main.py with the public GitHub repo key we received in the previous step:

# Add to main.py
AWS_ACCESS_KEY_ENCRYPTED = github.generate_encrypted_secret(
    github_public_key=github_repo_key['key'],
    secret_to_encrypt='YOUR_AWS_ACCESS_KEY'
)

AWS_SECRET_KEY_ENCRYPTED = github.generate_encrypted_secret(
    github_public_key=github_repo_key['key'],
    secret_to_encrypt='YOUR_AWS_SECRET_KEY'
)

12. Script for deploying repo secrets

Now that we have the encrypted secrets, we can write the function that deploys the secrets to the GitHub repo. Add this function to github.py:

def deploy_secrets_github_repo(owner, github_public_key_id, repo_name, secret_name, secret_value_encrypted, token):
    query_url = f"https://api.github.com/repos/{owner}/{repo_name}/actions/secrets/{secret_name}"
    headers = {'Accept': 'application/vnd.github.v3+json"', 'Authorization': f'token {token}'}
    params = {
        'owner': owner,
        'repo': repo_name,
        'secret_name': secret_name,
        'encrypted_value': secret_value_encrypted,
        'key_id': github_public_key_id
    }
    r = requests.put(query_url, headers=headers, data=json.dumps(params))
    return r.text

Finally, deploy the encrypted secrets to your GitHub repo by adding this snippet to main.py. Make sure to use the encrypted keys we previously created:

AWS_ACCESS_KEY_ID_SECRET = github.deploy_secrets_github_repo(
    owner=project_variables['owner'],
    github_public_key_id=github_repo_key['key_id'],
    repo_name=project_variables['repo_name'],
    secret_name='AWS_ACCESS_KEY_ID',
    secret_value_encrypted=AWS_ACCESS_KEY_ENCRYPTED,
    token=project_variables['token']
)

AWS_SECRET_ACCESS_KEY_SECRET = github.deploy_secrets_github_repo(
    owner=project_variables['owner'],
    github_public_key_id=github_repo_key['key_id'],
    repo_name=project_variables['repo_name'],
    secret_name='AWS_SECRET_ACCESS_KEY',
    secret_value_encrypted=AWS_SECRET_KEY_ENCRYPTED,
    token=project_variables['token']
)

Perfect, now you know how to add encrypted secrets to your GitHub repos:

13. Script for creating a develop branch

Before committing all the infrastructure, let's create a develop branch in your repo.

Why create a develop branch?

Develop branches are great for isolating new features and changes. By keeping your develop branch separate from your main branch, you can continue working on new features without worrying about merge conflicts. Additionally, having a separate develop branch makes it easy to track the progress of new features and changes.

Add the following function to github.py:

def add_new_branch_github_repo(branch, repo_path):
    """
    Add a new branch to an existing repo
    :return:
    """

    os.system(f"cd {repo_path} && git checkout -b {branch}")

Then call the function from your main.py file:

# Add to main.py
github.add_new_branch_github_repo(
    branch='develop', 
    repo_path=project_variables['repo_folder_path']
)

We're done with creating new branches:

14. Script for add, commit and push

Since we're on the develop branch, let's commit the rest of the code. Start by creating this function in github.py:

def commit_and_push_repo(branch, commit_comment, repo_path):
    """
    Add, commit and push a repo
    """

    os.system(f"cd {repo_path} && git add .")
    os.system(f"cd {repo_path} && git commit -m '{commit_comment}'")
    os.system(f"cd {repo_path} && git push -u origin {branch} ")

Then call the function from your main.py file:

# Add to main.py
github.commit_and_push_repo(branch="develop", 
                            commit_comment="Added the infrastructure",
                            repo_path=project_variables['repo_folder_path'])

Pushing to the develop branch will invoke the workflow for the infrastructure and run the command in your cloudformation.yml:

aws cloudformation deploy

Depending on your resources, this will take a while, especially if you are deploying CloudFront (CDNs).

Therefore we'll create a script that checks when the workflow is finished. We'll return to that shortly; let's first make the develop branch the default branch in the next step.

15. Script for making a branch default

Let's also create a script to make a specific branch the default. This part is optional, but I always make sure to have the develop branch as the default branch. This is to avoid accidentally pushing/breaking production code.

Add the following function to github.py:

def make_branch_default(branch, owner, repo_name, token):
    """
    Make a certain branch default
    """

    query_url = f"https://api.github.com/repos/{owner}/{repo_name}"
    headers = {
        'Accept': 'application/vnd.github.v3+json"', 
        'Authorization': f'token {token}'
    }
    payload = {
        "name": f"{repo_name}", 
        "default_branch": branch
    }
    r = requests.patch(
        query_url, 
        headers=headers, 
        data=json.dumps(payload)
    )

    return r.text

Then call the function in your main.py file:

# Add to main.py
github.make_branch_default(branch="develop",
                           owner=project_variables['owner'],
                           repo_name=project_variables['repo_name'],
                           token=project_variables['token'])

Check making a branch the default branch from our list:

16. Script for checking workflow status

Since the deployment of new backend resources can take several minutes, we'll need a script that listens for the status of the CI/CD workflow and only continues once the workflow is finished.

Add the following function to the github.py file:

import time

from tqdm import tqdm

def workflow_listener(owner, repo_name, token):
    time.sleep(10)
    retry = 0
    while True:
        retry += 1
        # List workflows
        query_url = f"https://api.github.com/repos/{owner}/{repo_name}/actions/runs"
        headers = {
            'Accept': 'application/vnd.github.v3+json"', 
            'Authorization': f'token {token}'
        }
        r = requests.get(query_url, headers=headers)

        workflow_response = json.loads(r.text)

        print("workflow_response: ", workflow_response)

        # Check the response, if no workflows at all
        if not workflow_response['workflow_runs']:
            print("No workflows at all")
            break

        if workflow_response['workflow_runs'][0]['status'] != 'completed':
            print(f"Still processing, retry {retry}")
            count_down = tqdm(range(60))
            for number in count_down:
                time.sleep(1)
                count_down.set_description(
                    f'Still workflow in process, retry {retry} in {number} {"second" if number == 1 else "seconds"}')
        else:

            if workflow_response['workflow_runs'][0]['conclusion'] == 'failure':
                print("Failure when checking the workflow")
                next_step = input("Do you want to continue Y or break n? Y/n: ")

                if next_step == 'n':
                    raise Exception("GitHub workflow error, the workflow had the conclusion failure")

            print("Done")
            pr_status_done = True
            break

Let's look at the function in more detail. This loop is basically waiting for the GitHub action response. While the response is other than failure or completed, the function sleeps/times out for 60 seconds before calling and retrying for a new response.

If the workflow fails, the loop breaks. I wrote this script since some workflows can sometimes run for up to 20 minutes (when deploying multiple CDNs or new certificates).

Using this function, the rest of the script will pause until all the changes are deployed, especially when you're waiting for a pull request to merge.

Finally call this listener from your main.py:

# Add to main.py
github.workflow_listener(owner=project_variables['owner'],
                         repo_name=project_variables['repo_name'],
                         token=project_variables['token'])

This function will also output a little smart progress bar that shows you a countdown of the 60-second timeout.

I added it so you know how many tries you had and how long the workflow listener has been running:

Progress bar

Now we have a CI/CD workflow listener:

17. Script for creating a pull request

Once the workflow is finished, you'll want to run a script that creates a pull request. Add the following function to your github.py file:

import subprocess

def create_pull_request(body, from_branch, repo_path, title, to_branch):
    """
    Create a new pull request
    """

    # Create pull request
    pull_request_response = subprocess.check_output(
        f"cd {repo_path} && gh pr create --title '{title}' --body '{body}' --base '{to_branch}' --head '{from_branch}'",
        shell=True)

    print("create_pull_request: ", pull_request_response)

    # Get the number of the pull request
    pull_request_number = pull_request_response.decode('utf-8').strip().split('/')[-1]

    return pull_request_number

This function returns a pull request number, which we'll use in a script later when we merge a successful pull request.

Call this new function from your main.py file:

# Add to main.py
pull_request_number = github.create_pull_request(
    body="First production deploy from template",
    from_branch="develop",
    repo_path=project_variables['repo_folder_path'],
    title="First prod deploy", 
    to_branch="main"
)

Check the box for creating a pull request:

18. Script for checking pull request status

While the pull request is running, we'll need another listener script similar to the one we created earlier. This one will wait for a finished pull request status.

Add the following function to your github.py file:

def pull_request_listener(repo_path):
    retry = 0
    while True:
        retry += 1
        pull_request_status = subprocess.check_output(f"cd {repo_path} && gh pr status",
                                                      shell=True)

        if 'Checks pending' in str(pull_request_status.decode("utf-8")):
            print(f"Still processing, retry {retry}")
            count_down = tqdm(range(60))
            for number in count_down:
                time.sleep(1)
                count_down.set_description(
                    f'Still processing, retry {retry} in {number} {"second" if number == 1 else "seconds"}')
        else:
            print("Done")
            pr_status_done = True
            break

Similar to the workflow status listener, this function will also display a little progress bar while running:

Progress bar

Now call the function from your main.py file:

# Add to main.py
github.pull_request_listener(repo_path=project_variables['repo_folder_path'])

We also have a pull request listener now:

As soon as the pull request is finished, we'll also need to add another listener for the workflow status. Do that by adding another snippet to your main.py file after the pull request listener:

# Add to main.py
github.workflow_listener(owner=project_variables['owner'],
                         repo_name=project_variables['repo_name'],
                         token=project_variables['token'])

19. Script for merging from one branch into another branch

This is the very last script we'll be running. Once the pull request is finished and the workflow status is completed, we're ready to merge the changes from the develop branch into the main branch.

Add the following function to your github.py file:

def merge_pull_request(pull_request_number, repo_path):
    """
    Merge a finished pull request
    """
    merge_response = subprocess.check_output(f"cd {repo_path} && gh pr merge {int(pull_request_number)} -m",
                                             shell=True)

    return merge_response.decode('utf-8')

This function uses the pull request number we saved in the earlier step. This number tells the API which pull request to actually merge.

Finally, call the function from your main.py file:

# Add to main.py
merge_status = github.merge_pull_request(pull_request_number=pull_request_number,
                                         repo_path=project_variables['repo_folder_path'])

End your main.py file with another listener for the workflow that you started with the merge:

# Add to main.py
github.workflow_listener(owner=project_variables['owner'],
                         repo_name=project_variables['repo_name'],
                         token=project_variables['token'])

Check the last checkbox, we're done with merging pull requests:

We're all set! 🎉

This is your complete infrastructure generation automation from start to finish. Here are all the 19 scripts we created to automate the entire backend infrastructure generation:

1. Script for defining project variables

2. Script for mono-repo folder structure creation

3. Script for creating folder structure for CI/CD in GitHub Actions

4. Script for creating the CI/CI workflow for infrastructure

5. Script for creating a gitignore file

6. Script for creating README file

7. Script for creating CloudFormation yml

8. Script for creating a new GitHub repo

9. Script for initiating and pushing to the repo

10. Script for fetching the public key of a repo

11. Script for generating encrypted AWS secrets

12. Script for deploying repo secrets

13. Script for creating a develop branch

14. Script for add, commit and push

15. Script for making a branch default

16. Script for checking workflow status

17. Script for creating a pull request

18. Script for checking pull request status

19. Script for merging from one branch into another branch

Next steps

1. Want to use the frontend you created to generate your own infrastructure as code files?

Get the yml file and the JSON file we created in the first step. Visit ➡️ https://d26k82x7lu48w8.cloudfront.net

2. Want the finished source code? You can get the entire repo here ➡️ https://norahsakal.gumroad.com/l/automate-infrastructure-generation

3. Do you need help with implementing your own infrastructure generation automation? Or do you have other questions?

I'm happy to help, don't hesitate to reach out ➡️ norah@quoter.se

1. Resources needed​

2. Building a frontend to pick resources​

- ⚛️ ReactJS​

3. Scripts to generate backend​

- 🐍 Python​

1. Resources needed​

Authorization | Cognito​

Storage | S3 buckets​

CDN | Cloudfront​

Database | DynamoDB​

2. Building a frontend to pick resources​

Create the Cloudformation YML​

3. Python scripts​

1. Script for defining project variables​

2. Script for mono-repo folder structure creation​

3. Script for creating folder structure for CI/CD in GitHub Actions​

4. Script for creating the CI/CI workflow for infrastructure​

5. Script for creating a gitignore file​

6. Script for creating README file​

7. Script for creating CloudFormation yml​

8. Script for creating a new GitHub repo​

9. Script for initiating and pushing to the repo​

10. Script for fetching the public key of a repo​

11. Script for generating encrypted AWS secrets​

12. Script for deploying repo secrets​

13. Script for creating a develop branch​

14. Script for add, commit and push​

15. Script for making a branch default​

16. Script for checking workflow status​

17. Script for creating a pull request​

18. Script for checking pull request status​

19. Script for merging from one branch into another branch​

1. Script for defining project variables​

2. Script for mono-repo folder structure creation​

3. Script for creating folder structure for CI/CD in GitHub Actions​

4. Script for creating the CI/CI workflow for infrastructure​

5. Script for creating a gitignore file​

6. Script for creating README file​

7. Script for creating CloudFormation yml​

8. Script for creating a new GitHub repo​

9. Script for initiating and pushing to the repo​

10. Script for fetching the public key of a repo​

11. Script for generating encrypted AWS secrets​

12. Script for deploying repo secrets​

13. Script for creating a develop branch​

14. Script for add, commit and push​

15. Script for making a branch default​

16. Script for checking workflow status​

17. Script for creating a pull request​

18. Script for checking pull request status​

19. Script for merging from one branch into another branch​

Next steps​

1. Want to use the frontend you created to generate your own infrastructure as code files?​

2. Want the finished source code? You can get the entire repo here ➡️ https://norahsakal.gumroad.com/l/automate-infrastructure-generation​

3. Do you need help with implementing your own infrastructure generation automation? Or do you have other questions?​

1. Resources needed

2. Building a frontend to pick resources

- ⚛️ ReactJS

3. Scripts to generate backend

- 🐍 Python

1. Resources needed

Authorization | Cognito

Storage | S3 buckets

CDN | Cloudfront

Database | DynamoDB

2. Building a frontend to pick resources

Create the Cloudformation YML

3. Python scripts

1. Script for defining project variables

2. Script for mono-repo folder structure creation

3. Script for creating folder structure for CI/CD in GitHub Actions

4. Script for creating the CI/CI workflow for infrastructure

5. Script for creating a gitignore file

6. Script for creating README file

7. Script for creating CloudFormation yml

8. Script for creating a new GitHub repo

9. Script for initiating and pushing to the repo

10. Script for fetching the public key of a repo

11. Script for generating encrypted AWS secrets

12. Script for deploying repo secrets

13. Script for creating a develop branch

14. Script for add, commit and push

15. Script for making a branch default

16. Script for checking workflow status

17. Script for creating a pull request

18. Script for checking pull request status

19. Script for merging from one branch into another branch

1. Script for defining project variables

2. Script for mono-repo folder structure creation

3. Script for creating folder structure for CI/CD in GitHub Actions

4. Script for creating the CI/CI workflow for infrastructure

5. Script for creating a gitignore file

6. Script for creating README file

7. Script for creating CloudFormation yml

8. Script for creating a new GitHub repo

9. Script for initiating and pushing to the repo

10. Script for fetching the public key of a repo

11. Script for generating encrypted AWS secrets

12. Script for deploying repo secrets

13. Script for creating a develop branch

14. Script for add, commit and push

15. Script for making a branch default

16. Script for checking workflow status

17. Script for creating a pull request

18. Script for checking pull request status

19. Script for merging from one branch into another branch

Next steps

1. Want to use the frontend you created to generate your own infrastructure as code files?

2. Want the finished source code? You can get the entire repo here ➡️ https://norahsakal.gumroad.com/l/automate-infrastructure-generation

3. Do you need help with implementing your own infrastructure generation automation? Or do you have other questions?