Image source: Unsplash

Previous Section: Getting Started with CVMFS: From Challenges to Practical Use Cases

CVMFS and DevOps

As we delve deeper into CVMFS and scientific data management, it becomes clear how essential DevOps practices are to keep everything running smoothly. In today’s fast-paced, data-driven environments, agility, efficiency, and scalability are crucial—and that’s exactly where DevOps shines.

By incorporating DevOps principles, we aim to simplify the setup and maintenance of our CVMFS infrastructure, automate complex deployment processes, and foster better collaboration between development and operations teams. This approach not only minimizes configuration errors but also allows for rapid updates, which are vital for managing large-scale data and distributed systems.

In short, DevOps enables us to handle CVMFS deployment with reliability and speed, transforming a potentially daunting setup into a streamlined, efficient workflow. In this post, we’ll explore how DevOps practices and tools can make your CVMFS setup robust, scalable, and easier to manage.

Prerequisites

Before diving into this section of our DevOps and CVMFS setup series, it’s important to have a foundational understanding of a few key technologies and concepts. Ensuring you’re comfortable with these areas will make the setup process smoother and help you get the most out of this guide.

Key Skills & Tools

1. Basic Understanding of Docker and Docker Compose
   • Familiarity with Docker commands and containerization concepts is essential, as Docker is central to creating our isolated, reproducible development environment. You should also know how to work with Docker Compose to orchestrate multi-container applications and manage shared configurations.
2. AWS CLI and EC2 Basics
   • Comfort with the AWS CLI for managing cloud resources directly from the command line will be invaluable. Additionally, understanding EC2 fundamentals (instances, key pairs, security groups) will help as we configure and deploy infrastructure in AWS.
3. Shell Scripting and Basic Linux Commands
   • This guide uses shell scripts to automate our environment setup. Familiarity with shell scripting basics (like defining functions, setting environment variables, and running commands) will be beneficial, as well as general comfort with Linux command-line usage.
4. Working with Environment Variables
   • Knowing how to manage environment variables is key, as we’ll securely store and pass AWS credentials, keys, and configuration values.

With these prerequisites, you’ll be well-prepared to follow along and leverage this DevOps workflow to manage a CVMFS-backed Kubernetes environment effectively. We will also assume that you have read the previous part of this blog, which talks about the details of CVMFS

DevOps Technologies

For our CVMFS setup, we’ll leverage several key DevOps technologies. While we won’t dive deeply into each tool here, understanding their roles is crucial as we progress. The main technologies we’ll use are:

• Docker: For containerizing our development environment, helping us manage dependencies and isolate our setup from the host system.
• Terraform: Our infrastructure-as-code tool that will automate and manage cloud resources efficiently.
• AWS: Our cloud provider, where we’ll host our infrastructure, enabling us to scale and manage deployments effectively.
• Kubernetes: To orchestrate our containers, ensuring smooth operation and scalability.

In this post, our focus will be on setting up a development environment using Docker, keeping our host system clean and preparing us for more complex deployments. By the end, you’ll be ready to tackle the rest of our DevOps workflow with confidence.

Setting Up Our Development Environment

Before we launch EC2 instances, we need a well-prepared development environment. At our company, we use Docker containers to ensure our setup is consistent and shareable, simplifying troubleshooting and making collaboration smoother.

Essential Tools

• Docker: To containerize our environment, keeping dependencies isolated and manageable.
• Terraform: To automate and manage our cloud resources with infrastructure as code.
• AWS CLI: For managing our cloud infrastructure directly from the command line.
• Lens (Optional): A Kubernetes dashboard for more intuitive cluster management.

Make sure Docker is installed on your host machine. There are plenty of resources to help guide you through the installation process. With Docker ready, we can move forward with setting up our environment.

Development Environment Provision

We’ll use a Docker container for our development environment, and to ensure consistency and portability, we’ll create a script to install all the necessary libraries and dependencies.

Why Use Docker Containers?

• Consistency: Docker provides an isolated environment, so everything is configured the same way, no matter where you run it.
• Portability: Containers are lightweight and can run anywhere Docker is supported, making collaboration easier.
• Simplified Dependency Management: Encapsulating dependencies within the container prevents conflicts with other software on your host machine.

Why Script the Provisioning Process?

Automating the setup process ensures:

• Reproducibility: A single command can recreate the environment identically, making onboarding and setup easy.
• Efficiency: Updates are straightforward, and everyone stays in sync with the latest configuration.
• Documentation: The script clearly lists which tools and libraries are needed and how they’re configured.

Development Environment Provision

In order to follow the theme of automation, we will create a script that will provision all of the setup we currently need for our development enviornment

#!/bin/bash

# provision-devenv.sh
# This script provisions a development environment by installing essential tools:
# - AWS CLI for managing AWS resources.
# - Terraform for infrastructure as code.
# - Base packages including curl, wget, and SSH client.

# Specify the AWS CLI and Terraform versions to be installed.
AWSCLI_VERSION=2.13.25
TERRAFORM_VERSION=1.6.1

# Function to install base packages needed for development and provisioning.
# This function:
# - Updates the system package list.
# - Installs commonly used packages like unzip, wget, curl, openssh-client, and rsync.
function install_base_packages(){
    sudo apt-get update  # Update package list.
    sudo apt-get install -y unzip wget curl openssh-client rsync  # Install essential packages.
}

# Function to install Terraform, a tool for provisioning infrastructure.
# This function:
# - Downloads the specified Terraform version.
# - Unzips and installs it in /usr/local/bin.
# - Cleans up temporary installation files.
function install_terraform() {
    cd /tmp  # Navigate to /tmp for installation files.
    wget https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip  # Download Terraform zip.
    unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip  # Unzip the downloaded file.
    sudo mv terraform /usr/local/bin/  # Move the Terraform binary to a directory in PATH.
    rm terraform_${TERRAFORM_VERSION}_linux_amd64.zip  # Remove the zip file after installation.
}

# Function to install the AWS CLI, a tool for managing AWS resources.
# This function:
# - Downloads the specified AWS CLI version.
# - Unzips and installs it.
# - Cleans up installation files once complete.
function install_aws_cli() {
    cd /tmp  # Navigate to /tmp for installation files.
    curl -fsSL "https://awscli.amazonaws.com/awscli-exe-linux-x86_64-$AWSCLI_VERSION.zip" -o "awscli.zip"  # Download AWS CLI zip.
    unzip awscli.zip > /dev/null  # Unzip the AWS CLI package quietly.
    sudo ./aws/install  # Install AWS CLI to the default location.
    cd /tmp  # Return to /tmp directory.
    rm -rf aws  # Remove AWS CLI installation files.
}

# Main function to execute all setup functions in sequence.
function main() {
    install_base_packages  # Install essential packages for environment setup.
    install_terraform      # Install the specified version of Terraform.
    install_aws_cli        # Install the specified version of AWS CLI.
}

# Run the main function to start the environment setup.
main

What This Script Does

• Install basic packages: Downloads basic packages that are required to perform required operations.
• Installs Terraform: Downloads and unzips the specified version of Terraform, then moves the binary to /usr/local/bin/.
• Installs AWS CLI: Downloads and unzips the AWS CLI installer, runs the installation, and cleans up temporary files.
• Automates Setup: Using this script ensures consistency and saves time, making the setup process efficient and repeatable.

Run this script inside your Docker container to provision your development environment.

AWS Credentials and Key Pair Setup

Now that we’ve installed the essentials, let’s configure our environment to connect to AWS and prepare it to manage EC2 instances. This involves setting up AWS credentials and obtaining an SSH key pair for secure access to our EC2 instances. Automating this process ensures that Terraform and other tools can interact seamlessly with AWS resources.

Step 1: Setting Up AWS Credentials

First, let’s automate the configuration of your AWS credentials using the following script:

#!/bin/bash

# This script configures AWS credentials and checks for the presence of a specified SSH key pair file.
# It ensures the necessary AWS environment variables are set and securely configures the permissions for the SSH key.

KEYNAME="YOUR_KEYPAIR_NAME"  # Replace with the name of your SSH key pair.

# Verify that the required AWS environment variables are set.
# These variables are essential for AWS CLI commands to authenticate and specify the default region.
if [ -z "$AWS_ACCESS_KEY" ] || [ -z "$AWS_SECRET_ACCESS_KEY" ] || [ -z "$AWS_DEFAULT_REGION" ]; then
    echo "Missing AWS environment variables: ACCESS_KEY, SECRET_KEY, or REGION"
    exit 1  # Exit the script if any variable is missing.
fi

# Check if the key pair file exists in the current directory.
# The script requires a key file matching the specified KEYNAME (e.g., YOUR_KEYPAIR_NAME.pem).
if [ ! -f "$KEYNAME.pem" ]; then
    echo "Key file $KEYNAME.pem not found in the current directory."
    exit 1  # Exit if the key file is not found.
fi

# Secure the key pair file by setting the permissions to read-only for the owner.
sudo chmod 400 $KEYNAME.pem  # Restrict permissions to prevent unauthorized access.

# Configure AWS CLI with the specified credentials and region.
# This ensures that the AWS CLI uses the provided credentials for subsequent commands.
aws configure set aws_access_key_id "$AWS_ACCESS_KEY"            # Set AWS access key ID.
aws configure set aws_secret_access_key "$AWS_SECRET_ACCESS_KEY"  # Set AWS secret access key.
aws configure set default.region "$AWS_DEFAULT_REGION"            # Set the default AWS region.
aws configure set output "$OUTPUT"                                # Set the output format (e.g., json, text).

Step 2: Obtaining an SSH Key Pair for EC2

To securely connect to your EC2 instances, you’ll need an SSH key pair. Here’s how to create and manage one:

1. Generate a Key Pair in AWS:
   • Log in to the AWS Management Console.
   • Navigate to EC2 > Key Pairs under the Network & Security section.
   • Click on Create Key Pair.
   • Provide a name for your key pair (e.g., my-ec2-keypair) and choose the .pem format.
   • Download the private key file (my-ec2-keypair.pem) and store it securely. This file will be used to SSH into your EC2 instances.

2. Set Permissions for Your Private Key:

   • On your local machine, run the following command to set the correct permissions for the private key file:

     chmod 400 my-ec2-keypair.pem
     

- This ensures that the private key is only accessible by you, which is crucial for security. 3. Update Your Docker Setup: - Make sure to copy the my-ec2-keypair.pem file into your project directory, and the Dockerfile will set the correct permissions for you.

Now that your AWS credentials and key pair are set up, let’s move on to setting up our Docker environment.

Docker Container Setup

Here’s the Dockerfile that defines our containerized development environment:

# Use Ubuntu 20.04 as the base image, providing a stable and widely used Linux environment.
FROM ubuntu:20.04

# Define build arguments for AWS credentials and region.
# These arguments allow you to pass in AWS credentials when building the Docker image.
ARG ACCESS_KEY
ARG SECRET_KEY
ARG REGION

# Set environment variables for AWS credentials, which are used by scripts in the container.
# These variables make the AWS CLI commands authenticate using the provided credentials and default region.
ENV AWS_ACCESS_KEY=$ACCESS_KEY
ENV AWS_SECRET_ACCESS_KEY=$SECRET_KEY
ENV AWS_DEFAULT_REGION=$REGION

# Set the working directory inside the container.
# All following commands will be executed from /workspace.
WORKDIR /workspace 

# Copy the contents of the current directory on the host to /workspace in the container.
COPY . /workspace/

# Update the package list and install sudo, allowing the container to use elevated privileges when required.
RUN apt-get update && apt-get install -y sudo

# Run the `provision.sh` script to configure the environment.
# This script typically sets up the development environment, installing necessary tools and packages.
RUN bash -c "./provision.sh"

# Run the `configure-aws.sh` script to set up AWS configuration within the container.
# This script likely uses the AWS credentials and region to authenticate the AWS CLI and set default settings.
RUN bash -c "./configure-aws.sh"

# Set the default command to launch a Bash shell, keeping the container running for further interaction.
CMD ["/bin/bash"]

Docker Compose

This is where Docker Compose truly shines, making it effortless to launch your development environment with just a simple docker-compose up.

Our docker-compose.yml file does more than just spin up a container. It passes crucial arguments to the Dockerfile and mounts our local directory into the container’s workspace. This setup means you can continue editing and working on your files locally, while the container takes care of running the necessary libraries and tools in a consistent, isolated environment.

Think of it as having a high-tech workshop at your disposal: you handle the design and construction on your local machine, while the container serves as a specialized toolbox, stocked with everything you need and always ready to assist, keeping your workflow efficient and streamlined.

# docker-compose.yaml

# Define the Docker Compose file version.
version: '3'

# Define the services to be run within this Docker Compose configuration.
services:
  devops:
    # Set the name of the container to 'devops' for easy identification and reference.
    container_name: devops

    # Build the container from a Dockerfile in the current context (directory).
    # The build arguments are passed in from environment variables to provide AWS credentials.
    build:
      context: .  # Use the current directory as the build context.
      args:
        ACCESS_KEY: "${ACCESS_KEY}"  # AWS access key from environment variable.
        SECRET_KEY: "${SECRET_KEY}"  # AWS secret access key from environment variable.
        REGION: "${REGION}"          # AWS default region from environment variable.

    # Mount the current directory on the host to /workspace in the container.
    # This allows files to be accessible and edited both inside and outside the container.
    volumes:
      - ".:/workspace"

    # Override the default entrypoint with `sh`, opening a shell session.
    entrypoint: "sh"

    # Keep the container running in interactive mode, useful for development and testing.
    tty: true            # Allocate a TTY to allow for an interactive shell.
    stdin_open: true     # Keep stdin open for interactive commands.

Adding the Finishing Touches: The .env File

Create a .env file to manage your AWS credentials securely, the docker compose file will automatically read them when you fire up the container:

ACCESS_KEY=[AWS_ACCESS_KEY_HERE]
SECRET_KEY=[AWS_SECRET_KEY_HERE]
REGION=[AWS_REGION_HERE]
OUTPUT=json

Running the Setup

Run the following command to start your container in detached-mode:

docker-compose up -d

This setup ensures a clean, portable, and efficient environment, ready for cloud-based tasks and further DevOps operations.

Remote Connect into the Container

To interact with your running container, you can use the following command:

docker exec -it devops-workspace bash

What’s Happening Here?

• docker exec: This command lets you run a command inside a running Docker container.
• it: These flags make the session interactive. i keeps the input stream open, and t allocates a pseudo-TTY, which allows you to work in the container’s command line as if you were directly accessing a regular shell environment.
• devops: This is the name of the container you want to access. It should match the container_name specified in your docker-compose.yml file.
• bash: This specifies that you want to open a Bash shell within the container.

How to Check if It’s Working

Once you’re inside the container, you can run a few commands to verify that your environment is set up correctly:

1. Check Installed Tools:

   • To check if Terraform is installed, run:

     terraform --version
     

- To check if the AWS CLI is installed, run:

bash aws --version

1. Verify Environment Variables:

   • Run the following command to make sure your AWS credentials and region are properly set:

     env | grep AWS
     

- This should display your AWS environment variables (e.g., AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_DEFAULT_REGION). 3. Inspect the Workspace: - Navigate to your /workspace directory to ensure your project files are accessible:

bash cd /workspace ls

- You should see all the files from your local directory mounted inside the container.

1. Inside of your working directory on your LOCAL machine, create an empty .txt file, now do a ls inside of your container and see if that file resides in there. If that works, then the volumes have mounted correctly!

If all these checks pass, your containerized development environment is set up and ready to use!

At this point in the guides, your working directory should look like the following.

├── devops/
    ├── .env
    ├── configure-aws.sh
    |── YOUR_KEY_PAIR.pem
    |── Dockerfile
    |── docker-compose.yml
    |── provision.sh
    ├── cvmfs-full-setup-basic/

In the next step of the guide, we learn about how we can use Terraform to be a powerful tool to create EC2 Instances, Security Groups and Provision our resources that will end up running our Kubernetes cluster.

Next Section: Terraform: What is it, how do we use it, our scripts to provision