LocalStack – Mock AWS cloud in your local development

AWS is one of leading cloud provider in current times. It provides various cloud services. Only way to develop applications using these services is to create free tier account in AWS. But AWS free tier account is valid for only one year. After one year only limited services are free. After free tier period you always run the risk of using some services which might be charged to your card. With LocalStack software you no longer need to depend on AWS for your experiments.It provides an easy-to-use test/mocking framework for developing cloud applications.

In this blog post, I will explain how to install LocalStack in your development machine and start using it to test your cloud based applications.

Requirements

• python(Python 2.x up to 3.8 supported) Download
• pip (python package manager) link
• Docker Download

Installation and Running

You can install LocalStack in your machine in 3 ways.

Using pip

Open your command prompt and run the following command

pip install localstackCode language: Python (python)

After installation, run the following command to start Local stack. By default, its get started inside Docker container

localstack startCode language: Python (python)

Using Docker command

Run the following docker command to install and start LocalStack.

docker run --rm -it -p 4566:4566 -p 4571:4571 localstack/localstackCode language: Python (python)

Using docker-compose

Download/copy the docker-compose.yml file from here and run the following command to install and start LocalStack.

docker-compose upCode language: Python (python)

LocalStack starts all its services on 4566 port.

You can check the status of LocalStack by pointing browser to http://localhost:4566

You can also check the health of each service by pointing the browser to http://localhost:4566/health

By default, LocalStack starts all the services. If you are looking to start only required services , you have to run the command like below. you have to specify the required service through the SERVICES option.

In Linux run below coomand

SERVICES=s3,kinesis docker-compose upCode language: Python (python)

In Windows run below command

set SERVICES=s3,kinesis && docker-compose upCode language: Python (python)

Creating and Accessing AWS Services

Once LocalStack is up and running, you can use the aws cli to create and access AWS services. Use the below command to install aws CLI, if not installed already.

pip install awscliCode language: Python (python)

Setting up aws region and credentials to run LocalStack

aws cli requires the region and the credentials to be set in order to run the aws commands. Create the default configuration & the credentials using below command on terminal.

aws configure --profile defaultCode language: Python (python)

It will ask for ‘AWS Access Key ID’ and ‘AWS Secret Access Key’ enter test for both of them. For all other options, take default options by pressing Enter key.

The above command creates config & credential files under <User-home>/.aws folder

Let’s create s3 buckets in LocalStack using aws cli. We need to point aws cli to LocalStack url to create infrastructure.

aws --endpoint-url=http://localhost:4566 s3 mb s3://mytestbucketCode language: Java (java)

Now let’s check whether s3 bucket is created or not using below command.

aws --endpoint-url=http://localhost:4566 s3api list-bucketsCode language: Java (java)

Using awslocal

If you observe above commands , with every command you need to specify the endpoint-url which is annoying. Fortunately there is way to avoid it using package called `awslocal`

awslocal is a thin CLI wrapper that runs commands directly against LocalStack (no need to specify --endpoint-url anymore).

you can run the s3 bucket list command like below

awslocal s3api list-bucketsCode language: Java (java)

Now lets look at the complex example to demonstrate the mocking of AWS infrastructure.

In this example I will demonstrate the vvent publshed to Kinesis stream is processed by AWS Lambda and logging the results to logs.

For AWS Lambda code which processes events from Kinesis stream , you can look at the one of my previous article and you can down load project code from GitHub

Check out the aws-kinesis-lambda code from GitHub and setup the project.

The project contains template.yaml SAM file which deploys and configures the lambda in AWS.

To deploy AWS lambda using SAM files , you need to install AWS SAM CLI. Please visit the link for download and install instructions.

Just like awslocal , we have wrapper around AWS SAM CLI called aws-sam-cli-local. install samlocal with below command.

pip install aws-sam-cli-localCode language: Python (python)

Deploying the AWS lambda.

Create the jar file by running following command from root of the Lambda project on commandline

mvn clean packageCode language: Python (python)

To deploy the lambda run below command from root of the project.

samlocal deploy --stack-name aws-kinesis-lambda --capabilities CAPABILITY_IAM --guided
Code language: Python (python)

The command will ask for some inputs, you can opt for default values. It will create all the required resources for Lambda.

Next we can confirm Lambda function deployment by running the below command.

awslocal lambda list-functions
Code language: Java (java)

Note down the function name highlighted in red colour^#1.We will use this later.

Now we will create the Kinesis stream.

awslocal kinesis create-stream --stream-name lambda-stream --shard-count 3
Code language: Python (python)

Now verify the stream creation

awslocal kinesis list-streams
Code language: Python (python)

We can see the stream details .

Note down the details highlighted in the red^#2. We will use this later

Now we need to create event source mapping between Kinesis stream and Lambda to give events as input to Lambda for processing.

awslocal lambda create-event-source-mapping --function-name aws-kinesis-lambda-lambda-e385e9de --batch-size 100 --starting-position AT_TIMESTAMP --starting-position-timestamp 1541139109 --event-source-arn arn:aws:kinesis:us-east-1:000000000000:stream/lambda-stream
Code language: Python (python)

Let’s put a event in the Kinesis stream.

awslocal kinesis put-record --stream-name lambda-stream --partition-key 000  --data "[{\"id\" : 1, \"name\" : \"Suresh\" , \"address\" : \"Hyderabad\" , \"salary\": 20} , {\"id\" : 2, \"name\" : \"Alex\" , \"address\" : \"Auckland\" , \"salary\": 40}]"
Code language: Python (python)

In our Lambda class after receiving the records we are just logging them. So lets check the logs and confirm that our Lambda received records.

First find out the available log groups.

awslocal logs describe-log-groupsCode language: Python (python)

Note down the logGroupName name from the output and use it in the below command to list all log streams for it.

awslocal logs describe-log-streams --log-group-name /aws/lambda/aws-kinesis-lambda-lambda-e385e9deCode language: Python (python)

Take the latest logStreamName from the output and use it in the below command to view the log statements.

awslocal logs get-log-events --log-group-name /aws/lambda/aws-kinesis-lambda-lambda-e385e9de --log-stream-name 2021/06/07/[LATEST]1a08613e
Code language: Python (python)

In the above output we can view log statements from Lambda and the values are matching with data put into Kinesis stream.

If you want to practice AWS CLI commands , LocalStack is the best option available in your local development.

We can also use LocalStack along with Java in integration testing of cloud application. I will cover few examples on this topic in the coming weeks.

References

https://lobster1234.github.io/2017/04/05/working-with-localstack-command-line
https://docs.aws.amazon.com/lambda/latest/dg/with-kinesis.html                                                               https://medium.com/@schogini/aws-kinesis-data-streams-a-tiny-cli-demo-9a20813bd082 https://github.com/localstack/localstack