How to debug Amazon ECS Containers

Here at Sumo we use Amazon ECS containers for a variety of different micro-tasks because being standard Docker containers they are very easy to develop and test with locally then simply use Docker push to run in the Amazon Cloud.

Generally, this goes off without a hitch, however, there are times when even though you tested locally, the container has issues on Amazon and you need to figure out why.

Usually, this is because of something different in the production environment, or IAM security policy issues.

Fortunately, when this happens it is pretty easy to get in and debug the issues once you know a few basic steps.

Find your container instance

To find your container instance, first log in to the Amazon Console and go to the Elastic Container Services section.

Once there, find and click on your Cluster and then switch to the Tasks tab and finally type in the input box to find the task you wish to debug.

Now if you click on the task to view the task information you should see the EC2 instance id in the Details tab. Click on this, or right click and open in a new tab and you will be taken directly to the EC2 instances page with the ECS host highlighted.

Simply select the EC2 instance and click the Connect button to obtain the SSH information for logging in to the selected instance.

Now that you have gathered all the required information we can begin with connecting to debug the instance.

Log in to your ECS host

Now that you have obtained the required details, drop yourself into a terminal or Putty and connect to the ECS host using your Amazon PEM key, then switch to the root user.

ssh -i ~/Documents/my-ec2-key.pem [email protected]
Last login: Wed Jul  4 23:30:54 2018 from 10.46.0.101

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2017.09-release-notes/
[ec2-user@ec2-51-203-125-105 ~]$ sudo su -
[root@ec2-51-203-125-105 ~]#

Analyze Docker Containers

Once you are in your ECS Docker host you can interact with Docker just as you would on any system. Let us start by listing the running Docker containers.

docker ps
CONTAINER ID        IMAGE                                                                                           COMMAND                  CREATED             STATUS              PORTS                              NAMES
cc2075dfd0a5        263289099978.dkr.ecr.us-west-2.amazonaws.com/sumo-microsumo-sumo-crawler:latest                 "pm2-runtime start..."   2 days ago          Up 2 days                                              ecs-microsumo-import-shopify-customers-11-sumo-microsumo-import-shopify-customers-84bbebf79eb3e6bff801
0d8fa64fdc16        amazon/amazon-ecs-agent:latest  

If you have lots of Docker containers you can look at the images, run commands and/or up times to determine where you which container you would like to debug.

Debug Docker containers

Once you are familiar with what containers you would like to interact with you have a few choices. You can attach to the docker container, simply view the logs, or execute another shell session.

Notice the usage of –sig-proxy=false for the docker attach command. This lets us issue a Ctrl-C to detach from the docker session without sending the signals through to the container and aborting the process.

docker attach --sig-proxy=false cc2075dfd0a5

^C
docker exec -it cc2075dfd0a5 /bin/sh
/code/src #
/code/src # exit
docker logs cc2075dfd0a5
 * Starting nxlog daemon...
   ...done.

These basic commands will handle most situations. If there is anything else you need to do just consult the standard Docker documentation or run docker help for a full list of commands.