{"action":"created","issue":{"url":"https://api.github.com/repos/01org/zephyr.js/issues/1046","repository_url":"https://api.github.com/repos/01org/zephyr.js","labels_url":"https://api.github.com/repos/01org/zephyr.js/issues/1046/labels{/name}","comments_url":"https://api.github.com/repos/01org/zephyr.js/issues/1046/comments","events_url":"https://api.github.com/repos/01org/zephyr.js/issues/1046/events","html_url":"https://github.com/01org/zephyr.js/issues/1046","id":222916427,"number":1046,"title":"RFC: New organization of tests","user":{"login":"grgustaf","id":664464,"avatar_url":"https://avatars1.githubusercontent.com/u/664464?v=3","gravatar_id":"","url":"https://api.github.com/users/grgustaf","html_url":"https://github.com/grgustaf","followers_url":"https://api.github.com/users/grgustaf/followers","following_url":"https://api.github.com/users/grgustaf/following{/other_user}","gists_url":"https://api.github.com/users/grgustaf/gists{/gist_id}","starred_url":"https://api.github.com/users/grgustaf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/grgustaf/subscriptions","organizations_url":"https://api.github.com/users/grgustaf/orgs","repos_url":"https://api.github.com/users/grgustaf/repos","events_url":"https://api.github.com/users/grgustaf/events{/privacy}","received_events_url":"https://api.github.com/users/grgustaf/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2017-04-20T01:02:41Z","updated_at":"2017-04-20T07:27:41Z","closed_at":null,"body":"We have a lot of test stuff scattered across the repo. We'd like to improve things and know exactly where a particular type of test belongs. We'd like to be able to more easily enumerate and automate running of tests. So we had some discussion on IRC and in person yesterday and I want to document a proposal based on the results, but we still welcome feedback and have open questions.\r\n\r\n### Current state\r\n| Path | Description |\r\n| --- | --- |\r\n| **`samples/tests/`** | Contains mostly informational tests written by developers |\r\n| **`scripts/trlite`** | Our sanity / CI test runner mostly has its tests embedded within |\r\n| **`scripts/`** | A couple source check tests are here too |\r\n| **`src/zjs_unit_tests.[ch]`** | Some C-only tests that get built into the linux executable |\r\n| **`src/zjs_test_promise.[ch]`** | A module to expose promise creation to JS for testing |\r\n| **`tests/`** | Automated (some manual) tests, mostly from QA, some from dev team |\r\n| **`tests/stress`** | Contains one test that runs indefinitely |\r\n\r\n### Considerations\r\n#### QA vs. Developer Tests\r\nWe have tests for QA and development purposes; should these be distinguished and how?\r\n\r\n#### Automated vs. Manual\r\nWe have both automated and manual tests. How should they be distinguished?\r\n\r\n#### JavaScript vs. C vs. Hybrid tests\r\nSome tests are pure JavaScript, some unit tests are pure C, and some need extra help from C, e.g. to isolate a particular module for good unit testing. Where should these go?\r\n\r\n#### Board-specific tests\r\nSome tests will only work on certain platforms. We can put tests in board-specific directories, but then what if a test works on a few different platforms; we wouldn't want to duplicate the test N times. So it might be better to have metadata specifying compatibility? Or just run the tests and mark \"allowed failures\" per platform somewhere. Also, why do the tests only work on certain platforms?\r\n* Platforms lack required hardware (e.g. BLE on k64f)\r\n* Platforms lack required Zephyr driver (e.g. AIO on k64f)\r\n* Other reasons? Do these reasons affect where the test should be and how it's designated?\r\n\r\n#### Hardware-dependent tests\r\nSome tests are automated once you have a certain hardware setup, such as a wire from IO2 to IO3. Should these be considered \"automated\" or \"manual\", or something else?\r\n\r\n#### Enhancements\r\nThere is plenty more to be done than just rearranging things. We need to be more clear about how tests are run. I recently added `make check` but it just calls `trlite -l`. What is really expected of it? We should maybe have a smart test runner that can run all the automated tests or named groups of them, maybe help you discover tests, or lead you through interactive testing by building and flashing for you with intervention as required.\r\n\r\nWe would also plan to experiment with mocking some Zephyr APIs in order to be able to regression test our API binding code automatically. Those would probably show up alongside or within modules, like `zjs_gpio_mock.c`.\r\n\r\n### Proposal\r\n| Path | Description |\r\n| ----- | --- |\r\n| **`samples/tests/`** | **Move** these into tests/manual. |\r\n| **`scripts/trlite`** | Leave this as the test runner but **move** embedded tests out into `tests/build`. |\r\n| **`scripts/`** | **Move** the source code check scripts to yet-to-be-named place for bash script tests that check the build/source. Leave this directory for other non-test scripts. |\r\n| **`src/zjs_unit_tests.[ch]`** | Move to `unittests/` with its own Makefile and executable rather than piggybacking on `jslinux` with `--unittests`. |\r\n| **`src/zjs_test_promise.[ch]`** | This is legitimate for a hybrid test's C portion to be here as a module. It will only be used by JS that requires the special test module, so it's automatically left out of any other build. So the tests that do use it don't need to be specially identified, they will just be in `tests/` or `tests/manual`. It's possible for such code to be added under `#ifdef` in the module it relates to; unclear which is better but this lets the source be less cluttered. Maybe `_test` should move to after the module name so they show up consecutively in `ls`. |\r\n| **`tests/`** | Core automated tests using the Assert.js module so their output can be checked for success with simple grep. However, drop test- prefix for tests as it is redundant with tests/ path. Move out manual tests. |\r\n| **`tests/<board>/`** | Board-specific tests, where &lt;board&gt; is a101, k64f, etc. |\r\n| **`tests/manual/`** | Tests that require manual intervention or (for now, at least) a specific hardware configuration such as external wiring. |\r\n| **`tests/scripts/`** | (Not sure of best name for this dir: bash, build, check, internal, sanity, source trlite?) It would contain bash-script tests pulled out of `trlite` and `scripts/` that check the source code or make sure code still builds in different configs. Maybe `check` since this is the stuff `make check` would run.\r\n| **`tests/stress`** | **Move** to `tests/manual`, maybe with -stress in filename. |\r\n| **`unittests/`** | **New** directory for C unit tests and a standalone executable that runs them. |\r\n"},"comment":{"url":"https://api.github.com/repos/01org/zephyr.js/issues/comments/295609529","html_url":"https://github.com/01org/zephyr.js/issues/1046#issuecomment-295609529","issue_url":"https://api.github.com/repos/01org/zephyr.js/issues/1046","id":295609529,"user":{"login":"pfalcon","id":500451,"avatar_url":"https://avatars0.githubusercontent.com/u/500451?v=3","gravatar_id":"","url":"https://api.github.com/users/pfalcon","html_url":"https://github.com/pfalcon","followers_url":"https://api.github.com/users/pfalcon/followers","following_url":"https://api.github.com/users/pfalcon/following{/other_user}","gists_url":"https://api.github.com/users/pfalcon/gists{/gist_id}","starred_url":"https://api.github.com/users/pfalcon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pfalcon/subscriptions","organizations_url":"https://api.github.com/users/pfalcon/orgs","repos_url":"https://api.github.com/users/pfalcon/repos","events_url":"https://api.github.com/users/pfalcon/events{/privacy}","received_events_url":"https://api.github.com/users/pfalcon/received_events","type":"User","site_admin":false},"created_at":"2017-04-20T07:27:41Z","updated_at":"2017-04-20T07:27:41Z","body":"> unittests/ \tNew directory for C unit tests and a standalone executable that runs them.\r\n\r\n+1. Except that you may want to make them sort together with other tests dir(s), so:\r\n\r\n* `tests-unit` - C-based unit tests\r\n"}}
{"actor":{"display_login":"pfalcon"}}