Debugging Containers

When containers don't behave as expected, you need effective debugging strategies. This lesson covers techniques for troubleshooting Docker containers at every level.

Common Issues

Symptom	Likely Causes
Container exits immediately	Missing entrypoint, crash on startup
Container won't start	Image not found, port conflict
Can't connect to service	Network issues, port not published
Application errors	Environment config, missing deps
Slow performance	Resource limits, volume mounts

Inspecting Containers

View Container Details

# Full container information
docker inspect mycontainer

# Specific fields
docker inspect -f '{{.State.Status}}' mycontainer
docker inspect -f '{{.Config.Env}}' mycontainer
docker inspect -f '{{.NetworkSettings.IPAddress}}' mycontainer
docker inspect -f '{{json .Mounts}}' mycontainer | jq

View Container State

# Current status
docker ps -a

# Detailed status
docker inspect -f '{{.State}}' mycontainer

# Exit code
docker inspect -f '{{.State.ExitCode}}' mycontainer

# Error message
docker inspect -f '{{.State.Error}}' mycontainer

Container Logs

Viewing Logs

# All logs
docker logs mycontainer

# Follow logs
docker logs -f mycontainer

# Last N lines
docker logs --tail 100 mycontainer

# With timestamps
docker logs -t mycontainer

# Since specific time
docker logs --since 10m mycontainer
docker logs --since 2024-01-15T10:00:00 mycontainer

# Combine options
docker logs -f -t --tail 50 mycontainer

Docker Compose Logs

# All services
docker compose logs

# Specific services
docker compose logs -f api db

# With timestamps and tail
docker compose logs -f -t --tail 100 api

Log Troubleshooting

# If logs are empty
docker inspect -f '{{.HostConfig.LogConfig}}' mycontainer

# Check log file directly
docker inspect -f '{{.LogPath}}' mycontainer
sudo cat $(docker inspect -f '{{.LogPath}}' mycontainer)

Interactive Debugging

Shell Access

# Bash (if available)
docker exec -it mycontainer bash

# Shell (more common in alpine)
docker exec -it mycontainer sh

# As root
docker exec -it -u root mycontainer sh

# With environment variable
docker exec -it -e DEBUG=true mycontainer sh

Running Debug Commands

# Check processes
docker exec mycontainer ps aux

# Check network
docker exec mycontainer netstat -tlnp
docker exec mycontainer ss -tlnp

# Check filesystem
docker exec mycontainer df -h
docker exec mycontainer ls -la /app

# Check environment
docker exec mycontainer env

# Check DNS
docker exec mycontainer nslookup other-container
docker exec mycontainer cat /etc/resolv.conf

Debugging Exited Containers

When a container exits before you can debug:

Check Exit Status

# View exit code
docker ps -a --filter name=mycontainer
# EXITED (1) = error, EXITED (0) = success

# Check logs before exit
docker logs mycontainer

Debug with Modified Entrypoint

# Override entrypoint to keep container running
docker run -it --entrypoint sh myimage

# Or use tail to keep alive
docker run -d --entrypoint tail myimage -f /dev/null
docker exec -it mycontainer sh

Create Debugging Container

# Run fresh container with same image
docker run -it --rm myimage sh

# Mount volumes from failed container
docker run -it --rm \
    --volumes-from failed_container \
    myimage sh

Network Debugging

Check Connectivity

# Test DNS resolution
docker exec mycontainer nslookup db
docker exec mycontainer ping -c 3 db

# Test TCP connection
docker exec mycontainer nc -zv db 5432

# Test HTTP endpoint
docker exec mycontainer curl -v http://api:3000/health
docker exec mycontainer wget -qO- http://api:3000/health

Inspect Networks

# List networks
docker network ls

# Inspect network
docker network inspect mynetwork

# Find container's IP
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' mycontainer

# Check which networks container is on
docker inspect -f '{{json .NetworkSettings.Networks}}' mycontainer | jq 'keys'

Network Debugging Container

# Run networking tools container
docker run -it --rm --network mynetwork nicolaka/netshoot

# Inside netshoot:
dig db
curl http://api:3000
tcpdump -i eth0

Resource Issues

Check Resource Usage

# Real-time stats
docker stats

# Specific container
docker stats mycontainer

# One-time snapshot
docker stats --no-stream

Identify Resource Problems

# Check limits
docker inspect -f '{{.HostConfig.Memory}}' mycontainer
docker inspect -f '{{.HostConfig.CpuShares}}' mycontainer

# OOM kills
docker inspect -f '{{.State.OOMKilled}}' mycontainer
dmesg | grep -i oom

Image Debugging

Analyze Layers

# View layer history
docker history myimage

# Full commands
docker history --no-trunc myimage

# Use dive for interactive analysis
dive myimage

Test Image Interactively

# Run with shell
docker run -it --rm myimage sh

# Run without default command
docker run -it --rm --entrypoint sh myimage

# Check what's in the image
docker run --rm myimage ls -la /app
docker run --rm myimage cat /app/package.json

Build Debugging

Debug Failed Builds

# Build with verbose output
docker build --progress=plain .

# Build specific stage
docker build --target builder -t myimage:debug .

# Run the debug image
docker run -it myimage:debug sh

BuildKit Debugging

# Disable BuildKit for clearer output
DOCKER_BUILDKIT=0 docker build .

# Or enable with plain progress
docker build --progress=plain .

Volume Debugging

Inspect Volumes

# List volumes
docker volume ls

# Inspect volume
docker volume inspect myvolume

# Check what's in a volume
docker run --rm -v myvolume:/data alpine ls -la /data

# Check permissions
docker run --rm -v myvolume:/data alpine stat /data

Fix Permission Issues

# Check container user
docker exec mycontainer whoami
docker exec mycontainer id

# Check file ownership
docker exec mycontainer ls -la /app/data

# Fix permissions (temporary)
docker exec -u root mycontainer chown -R node:node /app/data

Debugging Compose Services

# Validate compose file
docker compose config

# Check service configuration
docker compose config --services
docker compose config api

# Dry run
docker compose up --dry-run

# View generated container command
docker inspect -f '{{json .Config.Cmd}}' myproject_api_1

Debugging Tools

Useful Images

# Network debugging
docker run -it --rm nicolaka/netshoot

# Alpine with curl, wget
docker run -it --rm alpine/curl

# Full Linux environment
docker run -it --rm ubuntu

Installing Debug Tools

# In Alpine containers
docker exec -u root mycontainer apk add --no-cache curl strace

# In Debian/Ubuntu
docker exec -u root mycontainer apt-get update && apt-get install -y curl netcat

Debugging Checklist

1. Check container status: docker ps -a
2. Read the logs: docker logs mycontainer
3. Check exit code: docker inspect -f '{{.State.ExitCode}}'
4. Verify environment: docker exec mycontainer env
5. Test connectivity: docker exec mycontainer ping db
6. Check resources: docker stats mycontainer
7. Inspect configuration: docker inspect mycontainer
8. Shell into container: docker exec -it mycontainer sh

Key Takeaways

• Always check logs first with docker logs
• Use docker inspect for detailed container information
• Override entrypoint to debug containers that exit immediately
• Use network debugging containers like netshoot
• Check resource limits if containers are being killed
• Use docker stats to monitor resource usage
• Shell into containers to investigate issues interactively
• Keep debugging tools available in development images