Process killed by OOM killer

What this failure means

The process was terminated by the operating system’s out-of-memory (OOM) killer. Exit code 137 (128 + SIGKILL) is the canonical indicator.

Symptoms

Faultline looks for one or more of these log fragments:

exit code 137
OOMKilled
out of memory
oom killer
signal: killed
container killed
memory limit exceeded

Diagnosis

The process was terminated by the operating system’s out-of-memory (OOM) killer. Exit code 137 (128 + SIGKILL) is the canonical indicator.

Fix steps

1. Confirm the OOM kill. Exit code 137 means the process received SIGKILL:

   echo "Exit code: $?"            # run immediately after the failing step
   dmesg | grep -i 'oom\|kill' | tail -20   # on the host
   

   For Kubernetes: kubectl describe pod <pod> and look for OOMKilled in the Last State block.

2. Measure peak memory before increasing limits:

   docker stats --no-stream          # for running containers
   kubectl top pod                   # for Kubernetes pods
   /usr/bin/time -v <command>        # prints peak "Maximum resident set size"
   

3. Profile the process to find the largest allocators: go tool pprof, node --inspect + Chrome DevTools heap snapshot, valgrind --tool=massif, or py-spy dump --memory.

4. Reduce parallelism to lower peak memory use: fewer parallel test workers or build threads.

5. Check for memory leaks: loading large fixtures into memory, accumulating response bodies, or unbounded in-memory caches that grow over the test run.

6. Set explicit memory limits and requests in the deployment spec or CI runner tier so future OOM events fail fast with a clear reason.

Validation

• Monitor memory during the re-run: docker stats or kubectl top pod.
• Confirm the job completes with exit code 0, not 137.

Why it matters

The process consumed more memory than the CI runner’s available physical RAM or the container’s memory limit allowed. Large builds, test suites that load big fixtures, or memory leaks commonly trigger this.

Prevention

• Set an explicit memory limit that is slightly above normal usage to catch regressions early.
• Add a step that reports peak memory usage after each CI run.
• Define memory budgets in project benchmarks and track them over time.

Try it locally

/usr/bin/time -v <command>
/usr/bin/time -v <command>

How Faultline detects it

Use faultline explain oom-killed to see the full playbook.

faultline analyze build.log
faultline explain oom-killed

Generated from playbooks/bundled/log/runtime/oom-killed.yaml. Do not edit directly.