How not to use Athena !!!

For those who don't know, AWS Athena is a query service that makes it easy to read data stored in S3 bucket using SQL queries. It is optimized for querying huge amount of data and you don't even need to set up any infrastructure. But little did I know, it…

How to process nested arrays in json with Athena.

Suppose you are writing an application for a library. Instead of storing book inventory in traditional db, you decided to use s3. Each book record is converted to json, stringified, written to a file and stored in S3 as an object. To read this, you create tables in Athena and…

How to detect drift in AWS stack- part 2

In the first part, I had explained how to detect drift manually. Now we will see how to automate that. We are going to write a lambda function which can tell us whether there is any drift present in a given stack. There are boto3 api which helps to get…

Building data platform on AWS - part 1

You have joined this startup which wants to build a brand new,  scalable platform. With this it aims to reduces its operating cost and provides  better services to its customer. The platform in question here is the entire back end system to support  their products.  Since you don't want to…

How to detect drift in AWS stack- part 1

If you have deployed AWS stack using cloud formation template, there is a possibility that someone may have changed some of the properties on one or more resources without updating the CF template. As a result of this, your actual deployed resource is different from what is defined in the…

How to update athena table with glue

I was working on an application where data was stored in s3 bucket and athena was used to query this data. Since data was frequently updated (on an average, every 30 min or so), which wipes the data in s3 folder and writes new set of data,  I had to…