JUnit / SBT test suite reported one or more test failures

What this failure means

One or more JVM test cases failed. The JUnit runner or SBT reported the failure with a java.lang.AssertionError, sbt.TestsFailedException, or a Gradle/SBT test task summary showing failed tests.

Symptoms

Faultline looks for one or more of these log fragments:

java.lang.AssertionError
org.junit.Assert
org.junit.ComparisonFailure
AssertionFailedError
sbt.TestsFailedException
Tests unsuccessful
test tasks failed:
testOnly failed:

Diagnosis

A JUnit, SBT, or Gradle test run terminated with a non-zero exit code. Common output formats seen in CI:

Gradle / JUnit 4/5:

com.example.MyTest > shouldDoSomething FAILED
    java.lang.AssertionError: expected:<1> but was:<2>
        at org.junit.Assert.fail(Assert.java:88)

SBT:

[error] 1 of 17 test tasks failed:
[error] - it:testOnly failed: sbt.TestsFailedException: Tests unsuccessful

Common causes:

• Assertion failure — a test assertion (assertEquals, assertTrue, assertThat) did not hold; actual output diverged from expected.
• Memory/resource assertion — a test that tracks heap growth or object counts fails because GC did not reclaim expected memory in time (common in reactive/observable library tests like RxJava’s publishNoLeak).
• NullPointerException — a null reached a code path that did not expect it, typically caused by missing test setup or a changed API.
• Binary compatibility failure — a MiMa (mimaReportBinaryIssues) task detected that a public API signature changed in a binary-incompatible way.
• Data-dependent failure — the test relies on external state (database, filesystem, network) that is absent or inconsistent in CI.

Fix steps

1. Find the failing test in the output — look for FAILED lines or the stack trace that follows AssertionError.

2. Reproduce locally:

   # Gradle
   ./gradlew test --tests "com.example.MyTest.shouldDoSomething"
   
   # SBT
   sbt "testOnly com.example.MySpec"
   
   # Maven
   mvn -pl module test -Dtest=MyTest#testMethod
   

3. For AssertionError: X -> Y memory tests (e.g. RxJava publishNoLeak):

   • Run the test in isolation with -XX:+ExplicitGCInvokesConcurrent or add an explicit System.gc() call.
   • Check whether the test is inherently flaky (non-deterministic GC) and whether the project has a known skip annotation for such tests.

4. For sbt.TestsFailedException:

   • Run sbt testOnly *FailingSpec and examine the output.
   • Look for [error] lines above the exception for the actual test name and assertion message.

5. For MiMa binary compatibility (mimaReportBinaryIssues):

   • Run sbt mimaReportBinaryIssues locally.
   • Add a ProblemFilters.exclude[...] entry in build.sbt if the API change is intentional and the next release is major/minor.

6. Fix the root cause: update the implementation to match the test, or — if the test expectation is wrong — update the test and document the change.

Validation

• ./gradlew test (or sbt test / mvn test) exits zero with no failures.
• The specific failing test name no longer appears in FAILED output.

Why it matters

JVM test failures are deterministic once reproduced. The FAILED block names the test class, method, and exact assertion. A single failing test exits the JVM test runner with a non-zero code, blocking downstream steps.

Memory leak assertions (publishNoLeak) are the exception: they can be environment-sensitive because GC timing varies. Treat repeated flaky memory test failures as a signal to investigate actual retention, not just to skip the test.

Prevention

• Run ./gradlew test locally before pushing.
• Enable parallel test execution (maxParallelForks) with care — some tests share global state that causes flaky failures under parallelism.
• For MiMa, run sbt mimaReportBinaryIssues in the same PR that changes a public API.

Try it locally

./gradlew test
sbt test
mvn test
./gradlew test
sbt test

How Faultline detects it

Use faultline explain junit-test-failure to see the full playbook.

faultline analyze build.log
faultline explain junit-test-failure

Generated from playbooks/bundled/log/test/junit-test-failure.yaml. Do not edit directly.