I was recently setting up some monitoring of lambdas in Datadog. Part of the process requires you to set up a subscription filter in the log group to the Datadog lambda that ships the logs TO Datadog. We currently had some log groups that had been setup incorrectly that I was fixing when I ran into an issue in the AWS console. If a log group already had a subscription established you could not create a new one using the console.
Before we go any further let me just say that yes I know this should have been done in a CLI or even done way before hand in our Terraform…not the point here.
If I deleted the subscription from the log group and then tried to recreate it connecting to the correct Datadog lambda I would get an error that the statement id already existed in the function policy. You cant edit that policy for the function in the console to remove it by hand. Heres how I worked around the issue…
Using the AWS CLI I ran the following commands…
- DESCRIBE THE SUBSCRIPTION FILTER
aws logs describe-subscription-filters --log-group-name /aws/lambda/NAME_OF_LAMBDA_FUNCTION
- DELETE SUBSCRIPTION FILTER
aws logs delete-subscription-filter --log-group-name /aws/lambda/NAME_OF_LAMBDA_FUNCTION
- PUT SUBSCRIPTION FILTER
aws logs put-subscription-filter --log-group-name /aws/lambda/NAME_OF_LAMBDA_FUNCTION --filter-name LambdaStream_DataDogLogs --destination-arn arn:aws:lambda:us-east-1:YOUR_ACCOUNT_NUMBER:function:DataDogLogs --distribution ByLogStream --filter-pattern ""
Then in the console I went back into the Datadog lambda and double checked that the function policy was correct. I waited a couple of minutes and started seeing the logs for my lambda showing up in Datadog.