How to fix the non-idempotency issue for the transformers test suite (codecov issue)
Currently, codecov on PRs reports a pretty random information most of the time. If you look atthis analysisby Thomas Hu, he suggests that thetransformerstest suite isnot idempotent. i.e. each test run produces a coverage report that is different from the previous run, even if there were no changes in the code. (Idempotence: definition)There are several examples already in that ticket, but here is a recent “outrageous” report where a change in a pure doc file and no code change gets reported as a2.17% decrease in coverage.If you look at the reported percentages in the files, none of these make sense. Other than that the reported coverage from running the tests somehow wildly varies from run to run.screenshot639×1120 82.5 KBWhat I observed is that most of the time it’s the*_tf_*modules that are listed at the top of impacted files list, so I was thinking perhaps it has something to do specifically with TF, but it’s not always the case.If you have experience with this kind of situation, please, kindly share your insights and how this can be fixed.Thank you.
2020-08-21T18:43:00Z
Well, after an extensive testing I couldn’t find any symptoms of non-idempotency. It doesn’t mean it doesn’t exist, I just couldn’t reproduce it on my machine.There was one sub-issue of codecov generating an invalid report when it fails to find a code coverage report for the “base” it checks against. When this happens it goes looking for the nearest hash with the coverage report, which often leads to a report that is not representative of the true impact of the proposed PR, since it is no longer comparing the proposed code changes to the base against which it’d be applied.A fix that prevents generation of invalid reports for when the base is lacking has been applied:github.com/huggingface/transformersfix incorrect codecov reportshuggingface:master←stas00:patch-1opened03:23AM - 18 Aug 20 UTCstas00+4-0As discussed at https://github.com/huggingface/transformers/issues/6317 codecov…currently sends an invalid report when it fails to find a code coverage report for the base it checks against. When this happens it goes looking for the nearest hash with report, which often leads to a report that is not representative of the true impact of the proposed PR.

This PR fixes it by adding:

 `require_base: yes`        # don't report if there is no base coverage report

And then:

-  let's add this for clarity, this supposedly is already the default.

    `require_head: yes`        # don't report if there is no head coverage report 

- and perhaps no point reporting on doc changes as they don't make any difference to the coverage and the 0% change comment just generates noise:

   `require_changes: true`    # only comment if there was a change in coverage

These options are documented here: https://docs.codecov.io/docs/codecovyml-reference#commentBut looking at the recent PRs with no code change, the problem is still there. We still get coverage changes reported, when there are none:e.g.Codecov,Codecov