{"action":"created","issue":{"url":"https://api.github.com/repos/opendatakit/collect/issues/392","repository_url":"https://api.github.com/repos/opendatakit/collect","labels_url":"https://api.github.com/repos/opendatakit/collect/issues/392/labels{/name}","comments_url":"https://api.github.com/repos/opendatakit/collect/issues/392/comments","events_url":"https://api.github.com/repos/opendatakit/collect/issues/392/events","html_url":"https://github.com/opendatakit/collect/issues/392","id":205837328,"number":392,"title":"Start programmatic testing of core functionality","user":{"login":"lognaturel","id":967540,"avatar_url":"https://avatars.githubusercontent.com/u/967540?v=3","gravatar_id":"","url":"https://api.github.com/users/lognaturel","html_url":"https://github.com/lognaturel","followers_url":"https://api.github.com/users/lognaturel/followers","following_url":"https://api.github.com/users/lognaturel/following{/other_user}","gists_url":"https://api.github.com/users/lognaturel/gists{/gist_id}","starred_url":"https://api.github.com/users/lognaturel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lognaturel/subscriptions","organizations_url":"https://api.github.com/users/lognaturel/orgs","repos_url":"https://api.github.com/users/lognaturel/repos","events_url":"https://api.github.com/users/lognaturel/events{/privacy}","received_events_url":"https://api.github.com/users/lognaturel/received_events","type":"User","site_admin":false},"labels":[{"id":437069849,"url":"https://api.github.com/repos/opendatakit/collect/labels/enhancement","name":"enhancement","color":"5319e7","default":true},{"id":485736553,"url":"https://api.github.com/repos/opendatakit/collect/labels/high%20priority","name":"high priority","color":"b60205","default":false},{"id":474694259,"url":"https://api.github.com/repos/opendatakit/collect/labels/needs%20discussion","name":"needs discussion","color":"fbca04","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-02-07T09:52:13Z","updated_at":"2017-02-17T03:15:49Z","closed_at":null,"body":"As we start making bigger changes to the codebase, having some programmatic testing would help us avoid regressions. There's nothing worse than functionality that used to work suddenly breaking in a new release! (See #360 for a recent example 😧)\r\n\r\nThere was a [conversation on the dev mailing list](https://groups.google.com/forum/#!searchin/opendatakit-developers/testing/opendatakit-developers/JNwzuuIlhrg/4-f_6BTVBwAJ) about the topic.\r\n\r\n@srsudar recommended [Robolectric](http://robolectric.org/) and has some [example tests](https://github.com/srsudar/MamasDelRioAndroid/tree/master/app/src/test/java/org/mamasdelrio/android) in a Collect fork he worked on. Robolectric was added to the code base by #204\r\n\r\n@mitchellsundt pointed out that different tools are appropriate for different layers of tests. He says [instrumented unit tests](https://developer.android.com/training/testing/unit-testing/instrumented-unit-tests.html) have been very helpful for identifying regressions.\r\n\r\nClayton from Dimagi [explained their Android testing strategy](https://groups.google.com/d/msg/opendatakit-developers/JNwzuuIlhrg/-uRm1wYzAQAJ) using jUnit, Robolectric and [Calabash](http://calaba.sh/).\r\n\r\nLet's start with two specific things to test: getting a list of blank forms from an Aggregate instance that does have blank forms and downloading the first form in the list. What is the best approach for getting that basic functionality tested programmatically? How much should be stubbed, what's the best tool, etc."},"comment":{"url":"https://api.github.com/repos/opendatakit/collect/issues/comments/280538220","html_url":"https://github.com/opendatakit/collect/issues/392#issuecomment-280538220","issue_url":"https://api.github.com/repos/opendatakit/collect/issues/392","id":280538220,"user":{"login":"lognaturel","id":967540,"avatar_url":"https://avatars.githubusercontent.com/u/967540?v=3","gravatar_id":"","url":"https://api.github.com/users/lognaturel","html_url":"https://github.com/lognaturel","followers_url":"https://api.github.com/users/lognaturel/followers","following_url":"https://api.github.com/users/lognaturel/following{/other_user}","gists_url":"https://api.github.com/users/lognaturel/gists{/gist_id}","starred_url":"https://api.github.com/users/lognaturel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lognaturel/subscriptions","organizations_url":"https://api.github.com/users/lognaturel/orgs","repos_url":"https://api.github.com/users/lognaturel/repos","events_url":"https://api.github.com/users/lognaturel/events{/privacy}","received_events_url":"https://api.github.com/users/lognaturel/received_events","type":"User","site_admin":false},"created_at":"2017-02-17T03:15:49Z","updated_at":"2017-02-17T03:15:49Z","body":"@mitchellsundt @srsudar Thanks for your insights! @mitchellsundt do you have some reading material we could check out that advocates for a UI-testing-first approach? It's a very intuitively appealing idea since that kind of test would be most like what a user's experience is but I have not come across a project that emphasizes UI testing first. I understand what you're saying about using semantic elements (that's what @grzesiek2010 did with [his Espresso example](https://github.com/grzesiek2010/collect/tree/COLLECT-392_4)) but one of the eventual goals is to almost entirely replace the UI layer without maintaining its semantics.\r\n\r\nI think one of the things that makes this conversation challenging is that terms are not defined very precisely or consistently. \"Unit testing\" can mean something pedantic and far removed from reality and I certainly agree we should stay away from that. \"Coverage\" should not be a goal in and of itself. But unit testing at the class level with carefully considered mocks can add a lot of confidence in the well-functioning of parts of the code while also promoting much better design. I may have been quoting @srsudar there.\r\n\r\n@mitchellsundt said\r\n\r\n> Buggy code is where the effort should be focused, creating a nested web of tests to verify the code functions in isolation and in the context of the overall app. By starting with the app context, then elaborating the tests into an isolated state that mimics the app context, you can develop meaningful unit tests in the areas of code that are tricky and fragile.\r\n\r\nI very much agree with that and I believe we can do it without needing a UI testing layer (at first). We already know which parts of the code are brittle and mission critical (for example, we keep finding different creative ways to break the Google Sheets integration).\r\n\r\nAnother thing that occurs to me -- I don't think there's much debate around the tooling for unit testing in Android, right? Seems that jUnit 4 + Mockito (for ODK class mocks) + Robolectric (for Android class mocks) is the way to go. That's already all included and we have one glorious test from @lindsay-stevens. As we continue the discussion on which testing layers to focus most on and pick specific tools, can we also work on getting canonical examples of pure local unit tests, unit tests with Mockito stubs and unit tests with Robolectric stubs that we can discuss? I don't know if this is the right breakdown but it sounds like we agree that it would be useful to have solid examples of tests on different kinds of units. I think having something concrete to look at is always helpful.\r\n\r\n"}}
{"actor":{"display_login":"lognaturel"}}