If you are working with lambda functions / batch processes in your AWS stack, chances are that you may find it difficult to track errors in the log. Lambda / batch invocation may write events in different log stream and at the end of day, you have a lot of stream if you need to scan and see if there was any unexpected event.
Thankfully, you can automate this process with Cloud watch and SES.
For example, if you would like to get a notification every time there is an error logged in logs, you can create a cloud watch log subscription where you provide some keywords which you want to check and a lambda function which cloud watch invokes if it finds those keywords in any of the events. This invoked lambda can send an email to any email address you provide. With this flow in place, you get a real time alert whenever any error is logged.
Lets see some code.
For aws resources, I'm using cloudformation template. If you want to directly create these resources, please refer to the links at the end of article.
Creating Lambda function
LogSubscriptionProcessor: Type: AWS::Lambda::Function Properties: Description: processes event from cloudwatch log subscription Code: S3Bucket: my-bucket S3Key: LogSubscriptionProcessor.zip FunctionName: Log_Subscription_Processor Handler: index.handler Runtime: nodejs10.x MemorySize: 2048 Timeout: 120 Role: !GetAtt MyRole.Arn
Giving cloud watch permission to invoke lambda
Creating a subscription filter
SubscriptionFilter: Type: AWS::Logs::SubscriptionFilter Properties: DestinationArn: !GetAtt LogSubscriptionProcessor.Arn FilterPattern: "?error Error" LogGroupName: /aws/lambda/my-lambda DependsOn: AlertProessPermission
Policy to send e-mail
EmailPolicy: Type: AWS::IAM::ManagedPolicy Properties: PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - ses:SendEmail Resource: "*"
This policy you need to assign to the role under which lambda function is running.
This is all you need. In this set up , we have subscribed to cloud watch log group of lambda - my-lambda. So whenever this lambda writes any event which has key word 'error' of 'Error', cloud watch will invoke lambda function Log_Subscription_Processor. This lambda, after reading the event data, sends an email using amazon simple email service and you get the notification.
In next article, we will see how we can use slack to send those notifications.