Vishal Gaurav

Downloading AWS CodeWatch Log

VISHAL GAURAV's photo
VISHAL GAURAV

We run our build with AWS codebuild and our repository is in Bitbucket.

We have mono repo and the build we run inside code-build which is triggered on every PR creation.

Since we do a lot of stuff in build including container creation, and automated tests, it is sometimes required to analyze the entire log of one particular build.

AWS store the log in a JSON format which looks like this

{
'events': [
            'timestamp': 123,
            'message': 'string',
            'ingestionTime': 123
    ],
    'nextForwardToken': 'string',
    'nextBackwardToken': 'string'
 }

When you go to Cloud watch and try to analyze the log , you can not see the entire log. You can either search from the events or keep clicking load more button.

When you click on View entire log, it takes you to cloud watch and you still have to keep clicking Load more.

This is very inconvenient and sometime we would like to look at the entire log ( just the message ) and do a search like grep locally.

I looked at a tool called awslogs which is hosted in GitHub here https://github.com/jorgebastida/awslogs

It is an awesome tool, it allows me to search, and filter streams but doesn't let me have an entire log of a single stream.

At this point, I decided to write a small Python program using boto3 to get this job done for me.

The code is hosted here

https://github.com/vgaur/simple-aws-log-extract

It takes three parameter

  • group -> The AWS log group name

  • stream -> The AWS log stream inside the group

  • outfile -> The local file where the content will be downloaded.

python pull-aws-log.py --group XXX --stream YYY --outfile ZZZ

This does the job but its not handy.

So I updated script to delete and create the outfile. [ In future I can just take a location and create file based on stream name ]

Next I updated my zshrc file to make this a function

pullawslog() {
   python3 /Users/XXX/tools/awslog/pull-aws-log.py "$@"
}

I also updated my script to hide group name since I was only dealing with one group.

Now I can call the script like this

pullawslog --stream xx-ee-ss

Just one command and you can download entire stream log locally.

AWS#CloudWatchAWS CloudWatch