🛠️ Repo: Integration test failures on `main`

[JoshuaKGoldberg]

Running npm run test on main right now reports this summary:

Test Suites: 1 failed, 7 passed, 8 total
Tests:       4 failed, 62 passed, 66 total
Snapshots:   6 failed, 5 obsolete, 53 passed, 59 total

Full output including failing snapshot diffs

> all-contributors-bot@0.0.0-development test
> jest --coverage

 PASS  test/unit/helpers.test.js
 PASS  test/unit/comment-reply.test.js
 PASS  test/unit/setup-repository.test.js
 PASS  test/unit/config.test.js
WARNING :: Using the all-contributors node-api comes with zero guarantees of stability and may contain breaking changes without warning
 PASS  test/unit/parse-comment.test.js
 PASS  test/integration/smoke.test.js
 PASS  test/integration/installation.test.js
WARNING :: Using the all-contributors node-api comes with zero guarantees of stability and may contain breaking changes without warning
WARNING :: Using the all-contributors node-api comes with zero guarantees of stability and may contain breaking changes without warning
WARNING :: Using the all-contributors node-api comes with zero guarantees of stability and may contain breaking changes without warning
 FAIL  test/integration/issue_comment.test.js
  ● issue_comment event › Fail path, Unknown error (e.g. Network is dead, service down etc, our code is bad) crashes and sends error message

    expect(received).toMatchSnapshot(hint)

    Snapshot name: `issue_comment event Fail path, Unknown error (e.g. Network is dead, service down etc, our code is bad) crashes and sends error message: logs 1`

    - Snapshot  - 1
    + Received  + 4

    @@ -5,11 +5,14 @@
            "name": "issue_comment",
            "payload": Object {
              "action": "created",
              "comment": Object {
                "author_association": "MEMBER",
    -           "body": "@all-contributors please add jakebolam for code, doc and infra",
    +           "body": "@all-contributors please add jakebolam for code, doc and infra
    + 🤖 Beep boop! This comment was added automatically by [all-contributors-auto-action](https://github.com/marketplace/actions/all-contributors-auto-action).
    + Not all contributions can be detected from Git & GitHub alone. Please comment any missing contribution types this bot missed.
    + ...and of course, thank you for contributing! 💙",
                "created_at": "2019-01-10T08:36:52Z",
                "html_url": "https://github.com/all-contributors/all-contributors-bot/pull/1#issuecomment-453012966",
                "id": 453012966,
                "issue_url": "https://api.github.com/repos/all-contributors/all-contributors-bot/issues/1",
                "node_id": "MDEyOklzc3VlQ29tbWVudDQ1MzAxMjk2Ng==",

      341 |     delete output[0].request.headers["user-agent"];
      342 |
    > 343 |     expect(output).toMatchSnapshot("logs");
          |                    ^
      344 |   });
      345 |
      346 |   test("Happy path, add correct new contributor, but branch exists", async () => {

      at Object.<anonymous> (test/integration/issue_comment.test.js:343:20)

  ● issue_comment event › invalid .all-contributorsrc

    expect(received).toMatchSnapshot(hint)

    Snapshot name: `issue_comment event invalid .all-contributorsrc: request body 1`

    - Snapshot  - 1
    + Received  + 1

      Object {
        "body": "@jakebolam 

    - This project's configuration file has malformed JSON: .all-contributorsrc. Error:: Unexpected token o in JSON at position 1",
    + This project's configuration file has malformed JSON: .all-contributorsrc. Error:: Unexpected token 'o', \"nope\" is not valid JSON",
      }

      565 |         "/repos/all-contributors/all-contributors-bot/issues/1/comments",
      566 |         (body) => {
    > 567 |           expect(body).toMatchSnapshot("request body");
          |                        ^
      568 |           return true;
      569 |         }
      570 |       )

      at Object.<anonymous> (test/integration/issue_comment.test.js:567:24)
      at matchBody (node_modules/nock/lib/match_body.js:43:17)
      at Interceptor.match (node_modules/nock/lib/interceptor.js:366:17)
      at node_modules/nock/lib/intercepted_request_router.js:304:9
          at Array.find (<anonymous>)
      at InterceptedRequestRouter.startPlayback (node_modules/nock/lib/intercepted_request_router.js:303:45)
      at InterceptedRequestRouter.maybeStartPlayback (node_modules/nock/lib/intercepted_request_router.js:268:12)
      at InterceptedRequestRouter.connectSocket (node_modules/nock/lib/intercepted_request_router.js:140:12)
      at node_modules/nock/lib/intercepted_request_router.js:78:33

  ● issue_comment event › invalid .all-contributorsrc

    expect(received).toMatchSnapshot(hint)

    Snapshot name: `issue_comment event invalid .all-contributorsrc: logs 1`

    - Snapshot  - 1
    + Received  + 1

    @@ -2,9 +2,9 @@
        Object {
          "error": "AllContributorBotError",
          "id": "1",
          "isKnownError": true,
          "level": 30,
    -     "msg": "This project's configuration file has malformed JSON: .all-contributorsrc. Error:: Unexpected token o in JSON at position 1",
    +     "msg": "This project's configuration file has malformed JSON: .all-contributorsrc. Error:: Unexpected token 'o', \"nope\" is not valid JSON",
          "name": "event",
        },
      ]

      578 |
      579 |     expect(mock.activeMocks()).toStrictEqual([]);
    > 580 |     expect(output).toMatchSnapshot("logs");
          |                    ^
      581 |   });
      582 |
      583 |   test("User has no name and blog", async () => {

      at Object.<anonymous> (test/integration/issue_comment.test.js:580:20)

  ● issue_comment event › Unknown contribution

    expect(received).toMatchSnapshot(hint)

    Snapshot name: `issue_comment event Unknown contribution: request body 1`

    - Snapshot  - 1
    + Received  + 1

      Object {
        "body": "@jakebolam 

      I couldn't determine any contributions to add, did you specify any contributions?
    -           Please make sure to use [valid contribution names](https://allcontributors.org/docs/en/emoji-key).",
    +         Please make sure to use [valid contribution names](https://allcontributors.org/docs/en/emoji-key).",
      }

      734 |         "/repos/all-contributors/all-contributors-bot/issues/1/comments",
      735 |         (body) => {
    > 736 |           expect(body).toMatchSnapshot("request body");
          |                        ^
      737 |           return true;
      738 |         }
      739 |       )

      at Object.<anonymous> (test/integration/issue_comment.test.js:736:24)
      at matchBody (node_modules/nock/lib/match_body.js:43:17)
      at Interceptor.match (node_modules/nock/lib/interceptor.js:366:17)
      at node_modules/nock/lib/intercepted_request_router.js:304:9
          at Array.find (<anonymous>)
      at InterceptedRequestRouter.startPlayback (node_modules/nock/lib/intercepted_request_router.js:303:45)
      at InterceptedRequestRouter.maybeStartPlayback (node_modules/nock/lib/intercepted_request_router.js:268:12)
      at InterceptedRequestRouter.connectSocket (node_modules/nock/lib/intercepted_request_router.js:140:12)
      at node_modules/nock/lib/intercepted_request_router.js:78:33

  ● issue_comment event › Unknown contribution

    expect(received).toMatchSnapshot(hint)

    Snapshot name: `issue_comment event Unknown contribution: logs 1`

    - Snapshot  - 1
    + Received  + 1

    @@ -8,11 +8,11 @@
          "createdBy": 3534236,
          "createdByLogin": "jakebolam",
          "createdByType": "user",
          "id": "1",
          "level": 30,
    -     "msg": "No contributions",
    +     "msg": "No contributions for jakebolam",
          "name": "event",
          "private": false,
          "repository": 164268911,
          "success": false,
          "who": "jakebolam",

      747 |
      748 |     expect(mock.activeMocks()).toStrictEqual([]);
    > 749 |     expect(output).toMatchSnapshot("logs");
          |                    ^
      750 |   });
      751 |
      752 |   test("Comment is by app itself", async () => {

      at Object.<anonymous> (test/integration/issue_comment.test.js:749:20)

  ● issue_comment event › Happy path, add correct new multiple contributors

    expect(received).toStrictEqual(expected) // deep equality

    - Expected  -  1
    + Received  + 11

    - Array []
    + Array [
    +   "GET https://api.github.com:443/repos/all-contributors/all-contributors-bot/git/ref/heads%2Fall-contributors%2Fadd-gr2m",
    +   "GET https://api.github.com:443/repos/all-contributors/all-contributors-bot/contents/.all-contributorsrc",
    +   "GET https://api.github.com:443/users/gr2m",
    +   "GET https://api.github.com:443/repos/all-contributors/all-contributors-bot/contents/README.md",
    +   "GET https://api.github.com:443/repos/all-contributors/all-contributors-bot/git/ref/heads%2Fmaster",
    +   "POST https://api.github.com:443/repos/all-contributors/all-contributors-bot/git/refs",
    +   "PUT https://api.github.com:443/repos/all-contributors/all-contributors-bot/contents/.all-contributorsrc",
    +   "PUT https://api.github.com:443/repos/all-contributors/all-contributors-bot/contents/README.md",
    +   "POST https://api.github.com:443/repos/all-contributors/all-contributors-bot/pulls",
    + ]

      851 |     });
      852 |
    > 853 |     expect(mock.activeMocks()).toStrictEqual([]);
          |                                ^
      854 |     expect(output).toMatchSnapshot("logs");
      855 |   });
      856 | });

      at Object.<anonymous> (test/integration/issue_comment.test.js:853:32)

  ● issue_comment event › Happy path, add correct new multiple contributors

    expect(received).toMatchSnapshot(hint)

    Snapshot name: `issue_comment event Happy path, add correct new multiple contributors: request body 5`

    - Snapshot  - 2
    + Received  + 3

      Object {
    -   "ref": "refs/heads/all-contributors/add-gr2m",
    -   "sha": "aa218f56b14c9653891f9e74264a383fa43fefbd",
    +   "body": "@jakebolam 
    +
    + I've put up [a pull request](https://github.com/all-contributors/all-contributors-bot/pull/1347) to add @tenshiamd! :tada:",
      }

      839 |         "/repos/all-contributors/all-contributors-bot/issues/1/comments",
      840 |         (body) => {
    > 841 |           expect(body).toMatchSnapshot("request body");
          |                        ^
      842 |           return true;
      843 |         }
      844 |       )

      at Object.<anonymous> (test/integration/issue_comment.test.js:841:24)
      at matchBody (node_modules/nock/lib/match_body.js:43:17)
      at Interceptor.match (node_modules/nock/lib/interceptor.js:366:17)
      at node_modules/nock/lib/intercepted_request_router.js:304:9
          at Array.find (<anonymous>)
      at InterceptedRequestRouter.startPlayback (node_modules/nock/lib/intercepted_request_router.js:303:45)
      at InterceptedRequestRouter.maybeStartPlayback (node_modules/nock/lib/intercepted_request_router.js:268:12)
      at InterceptedRequestRouter.connectSocket (node_modules/nock/lib/intercepted_request_router.js:140:12)
      at node_modules/nock/lib/intercepted_request_router.js:78:33

 › 6 snapshots failed.
 › 5 snapshots obsolete.
   • issue_comment event Happy path, add correct new multiple contributors: logs 1
   • issue_comment event Happy path, add correct new multiple contributors: request body 6
   • issue_comment event Happy path, add correct new multiple contributors: request body 7
   • issue_comment event Happy path, add correct new multiple contributors: request body 8
   • issue_comment event Happy path, add correct new multiple contributors: request body 9
----------------------------------|---------|----------|---------|---------|-------------------
File                              | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
----------------------------------|---------|----------|---------|---------|-------------------
All files                         |     100 |      100 |     100 |     100 |                   
 all-contributors-app             |     100 |      100 |     100 |     100 |                   
  app.js                          |     100 |      100 |     100 |     100 |                   
 all-contributors-app/lib         |     100 |      100 |     100 |     100 |                   
  add-contributor.js              |     100 |      100 |     100 |     100 |                   
  get-config.js                   |     100 |      100 |     100 |     100 |                   
  get-user-details.js             |     100 |      100 |     100 |     100 |                   
  is-message-by-app.js            |     100 |      100 |     100 |     100 |                   
  is-message-for-app.js           |     100 |      100 |     100 |     100 |                   
  parse-comment.js                |     100 |      100 |     100 |     100 |                   
  process-issue-comment.js        |     100 |      100 |     100 |     100 |                   
  setup-repository.js             |     100 |      100 |     100 |     100 |                   
  to-safe-git-reference-name.js   |     100 |      100 |     100 |     100 |                   
 all-contributors-app/lib/modules |     100 |      100 |     100 |     100 |                   
  comment-reply.js                |     100 |      100 |     100 |     100 |                   
  config.js                       |     100 |      100 |     100 |     100 |                   
  content-files.js                |     100 |      100 |     100 |     100 |                   
  errors.js                       |     100 |      100 |     100 |     100 |                   
  helpers.js                      |     100 |      100 |     100 |     100 |                   
  repository.js                   |     100 |      100 |     100 |     100 |                   
----------------------------------|---------|----------|---------|---------|-------------------
Snapshot Summary
 › 6 snapshots failed from 1 test suite. Inspect your code changes or run `npm test -- -u` to update them.
 › 5 snapshots obsolete from 1 test suite. To remove them all, run `npm test -- -u`.
   ↳ test/integration/issue_comment.test.js
       • issue_comment event Happy path, add correct new multiple contributors: logs 1
       • issue_comment event Happy path, add correct new multiple contributors: request body 6
       • issue_comment event Happy path, add correct new multiple contributors: request body 7
       • issue_comment event Happy path, add correct new multiple contributors: request body 8
       • issue_comment event Happy path, add correct new multiple contributors: request body 9

Test Suites: 1 failed, 7 passed, 8 total
Tests:       4 failed, 62 passed, 66 total
Snapshots:   6 failed, 5 obsolete, 53 passed, 59 total
Time:        1.339 s, estimated 2 s
Ran all test suites.

Running npm run test -- -u updates all the failing snapshots. But that still leaves one non-trivial test failure I haven't had time to debug:

 FAIL  test/integration/issue_comment.test.js
  ● issue_comment event › Happy path, add correct new multiple contributors

    expect(received).toStrictEqual(expected) // deep equality

    - Expected  -  1
    + Received  + 11

    - Array []
    + Array [
    +   "GET https://api.github.com:443/repos/all-contributors/all-contributors-bot/git/ref/heads%2Fall-contributors%2Fadd-gr2m",
    +   "GET https://api.github.com:443/repos/all-contributors/all-contributors-bot/contents/.all-contributorsrc",
    +   "GET https://api.github.com:443/users/gr2m",
    +   "GET https://api.github.com:443/repos/all-contributors/all-contributors-bot/contents/README.md",
    +   "GET https://api.github.com:443/repos/all-contributors/all-contributors-bot/git/ref/heads%2Fmaster",
    +   "POST https://api.github.com:443/repos/all-contributors/all-contributors-bot/git/refs",
    +   "PUT https://api.github.com:443/repos/all-contributors/all-contributors-bot/contents/.all-contributorsrc",
    +   "PUT https://api.github.com:443/repos/all-contributors/all-contributors-bot/contents/README.md",
    +   "POST https://api.github.com:443/repos/all-contributors/all-contributors-bot/pulls",
    + ]

      851 |     });
      852 |
    > 853 |     expect(mock.activeMocks()).toStrictEqual([]);
          |                                ^
      854 |     expect(output).toMatchSnapshot("logs");
      855 |   });
      856 | });

      at Object.<anonymous> (test/integration/issue_comment.test.js:853:32)

[lwasser]

This is definitely a priority item for us to work on, as we can't deploy or merge anything if tests are broken. Thank you for opening!

With the docs, we created milestones - I wonder if we want to do that here as well. We might want to have a list of items like

1. Document infrastructure used to deploy the app
2. Fix the tests
3. Fix the vercel deployment.

My guess is that under each of the above items, we will find sub-issues that we can open as we discover them. That is what happened with the documentation. It was a big can of worms that exploded 🪱🪱🪱🪱🪱🪱🪱 🙃

[lwasser]

Are you planning to / working on this @JoshuaKGoldberg ? If so, let's assign it to you so folks know what is being worked on!

[lwasser]

@all-contributors please add @JoshuaKGoldberg for tests, bug

[allcontributors]

@lwasser

I've put up a pull request to add @JoshuaKGoldberg! 🎉

[JoshuaKGoldberg]

Are you planning to / working on this

Not soon 😞. I haven't had the time yet to deep dive into the app or back into the CLI, and probably won't for at least another week or two. TBD.

If someone else wants to tackle this, that'd be lovely.