Merged test coverage reports

[maxbebaasuro]

Description

I'd like to have one unit test coverage report across all projects. Basically the combined version of all the reports in the coverage directory. Not sure if this can be accomplished by nx as you could have different coverage report formats in the same workspace.

Motivation

It would simplify integration with azure devops or compodoc a lot as both expect a single report.

Suggested Implementation

An optional flag could output a combined report
nx run-many --target=test --all --parallel --coverage --combine-coverage

Alternate Implementations

Nx could allow to add a post-task that runs after generating the tests. Similar to DependsOn but after running a task. This way you could do all sorts of merging, tidy up or other jobs after any task execution.

[maxbebaasuro]

I was again thinking about it. Instead of making this so complicated. Wouldn't it make sense to have workspace tasks? Basically something like this from project.json but on workspace.json level,:

"targets": {
    "whatever": {},
    "foo": {}
}

That way people could use for example nx:run-commands on a workspace level.

[AgentEnder]

I'm going to flag this as related to testing tools for obvious reasons, but the overall viability of this may be a bit harder than one might imagine. Flags being passed into the executors are passed in by a per-executor basis. Executor runs are silo'd off for the most part, each instance of @nrwl/jest:jest as an example doesn't know about the other instances running when using affected or run-many. In the case of DTE, these would actually be distributed across machines, making it less feasible there too.

One option that may work for something like this, would be to have your coverage root directory setup as a project. In this case, it may looks something like this:

File Tree

my-workspace/
├─ node_modules/
├─ coverage/
│  ├─ lib-a/
│  │  ├─ coverage-report.json
│  ├─ lib-b/
│  │  ├─ coverage-report.json
│  ├─ project.json
├─ projects/
│  ├─ lib-a/
│  │  ├─ index.spec.ts
│  │  ├─ index.ts
│  │  ├─ project.json
│  ├─ lib-b/
│  │  ├─ index-b.ts
│  │  ├─ index-b.spec.ts
│  │  ├─ project.json
├─ .gitignore
├─ package.json
├─ nx.json
├─ README.md

.gitignore

coverage/*/coverage-report.json // Note this can't ignore the whole coverage directory

coverage/project.json

{
  "targets": {
    "combine-reports": { // NOTE: This **_CANNOT_** be cache-able, since its source files are in gitignore and the hash would not change. Test could still be cacheable though.
      "executor": "nx:run-commands",
      "options": {
        "command": "node ../tools/scripts/combine-coverage-reports"
      },
      "dependsOn": [{
        "target": "test",
        "project": "dependencies"
      }],
      "implicitDependencies": ["lib-a", "lib-b"]
    }
  }
}

Workspace level targets are discussed here: #10002. You are correct in that they could also be used for a similar purpose.

[comcaptain]

I'm going to flag this as related to testing tools for obvious reasons, but the overall viability of this may be a bit harder than one might imagine. Flags being passed into the executors are passed in by a per-executor basis. Executor runs are silo'd off for the most part, each instance of @nrwl/jest:jest as an example doesn't know about the other instances running when using affected or run-many. In the case of DTE, these would actually be distributed across machines, making it less feasible there too.

One option that may work for something like this, would be to have your coverage root directory setup as a project. In this case, it may looks something like this:

File Tree

my-workspace/
├─ node_modules/
├─ coverage/
│  ├─ lib-a/
│  │  ├─ coverage-report.json
│  ├─ lib-b/
│  │  ├─ coverage-report.json
│  ├─ project.json
├─ projects/
│  ├─ lib-a/
│  │  ├─ index.spec.ts
│  │  ├─ index.ts
│  │  ├─ project.json
│  ├─ lib-b/
│  │  ├─ index-b.ts
│  │  ├─ index-b.spec.ts
│  │  ├─ project.json
├─ .gitignore
├─ package.json
├─ nx.json
├─ README.md

.gitignore

coverage/*/coverage-report.json // Note this can't ignore the whole coverage directory

coverage/project.json

{
  "targets": {
    "combine-reports": { // NOTE: This **_CANNOT_** be cache-able, since its source files are in gitignore and the hash would not change. Test could still be cacheable though.
      "executor": "nx:run-commands",
      "options": {
        "command": "node ../tools/scripts/combine-coverage-reports"
      },
      "dependsOn": [{
        "target": "test",
        "project": "dependencies"
      }],
      "implicitDependencies": ["lib-a", "lib-b"]
    }
  }
}

Workspace level targets are discussed here: #10002. You are correct in that they could also be used for a similar purpose.

Can you give a bit more detail about this solution? e.g. cli command to create the coverage project, content of ../tools/scripts/combine-coverage-reports. It would be great if there is sample repo

[zzau13]

Could be more interesting can get jest --outputFile from root directory and use ArtiomTr/jest-coverage-report-action in github actions for report it.

[Lonli-Lokli]

@AgentEnder workspace level targets has been landed here, can this task be implemented too?

[github-actions[bot]]

This issue has been automatically marked as stale because it hasn't had any recent activity. It will be closed in 14 days if no further activity occurs.
If we missed this issue please reply to keep it active.
Thanks for being a part of the Nx community! 🙏

[jensbodal]

I might try giving something like this a shot: https://stackoverflow.com/a/65902458

Though unclear if should be using nyc now, I don't know the status of the istanbuljs ecosystem.

Currently for coverage I run a separate script in ci that just runs jest directly as a multi project suite but it's extremely slow.

[konstantinschuette]

As we now migrated to Jest it would be pretty useful to have it. Are there any plans to integrated this?

[Kepro]

I would suggest migrate away from jest

[konstantinschuette]

I would suggest migrate away from jest

Sure, now that's a valid point after Angular uses Vitest by default. But does the test runner have any impact on this issue? Isn't that on the side of NX?