{"action":"created","issue":{"url":"https://api.github.com/repos/dotnet/BenchmarkDotNet/issues/566","repository_url":"https://api.github.com/repos/dotnet/BenchmarkDotNet","labels_url":"https://api.github.com/repos/dotnet/BenchmarkDotNet/issues/566/labels{/name}","comments_url":"https://api.github.com/repos/dotnet/BenchmarkDotNet/issues/566/comments","events_url":"https://api.github.com/repos/dotnet/BenchmarkDotNet/issues/566/events","html_url":"https://github.com/dotnet/BenchmarkDotNet/issues/566","id":264406065,"number":566,"title":"Strategy for making sure that microbenchmarks are not broken for regular CI builds","user":{"login":"pakrym","id":1697911,"avatar_url":"https://avatars0.githubusercontent.com/u/1697911?v=4","gravatar_id":"","url":"https://api.github.com/users/pakrym","html_url":"https://github.com/pakrym","followers_url":"https://api.github.com/users/pakrym/followers","following_url":"https://api.github.com/users/pakrym/following{/other_user}","gists_url":"https://api.github.com/users/pakrym/gists{/gist_id}","starred_url":"https://api.github.com/users/pakrym/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pakrym/subscriptions","organizations_url":"https://api.github.com/users/pakrym/orgs","repos_url":"https://api.github.com/users/pakrym/repos","events_url":"https://api.github.com/users/pakrym/events{/privacy}","received_events_url":"https://api.github.com/users/pakrym/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-10-10T23:13:00Z","updated_at":"2017-10-11T15:30:54Z","closed_at":null,"author_association":"NONE","body":"We don't run microbenchmarks during every build/test run because we want to keep them fast. The consequence is that while we still build microbenchmarks projects we don't run them and don't notice when they start failing at runtime.\r\n\r\nIs there any guideline on how to properly integrate running couple iterations of every microbenchmark into CI pipeline to validate that they are functional in every build and do it reasonably quickly?\r\n\r\nHaving something like `dotnet run -validate` when using `BenchmarkSwitcher` would be ideal."},"comment":{"url":"https://api.github.com/repos/dotnet/BenchmarkDotNet/issues/comments/335850626","html_url":"https://github.com/dotnet/BenchmarkDotNet/issues/566#issuecomment-335850626","issue_url":"https://api.github.com/repos/dotnet/BenchmarkDotNet/issues/566","id":335850626,"user":{"login":"AndreyAkinshin","id":2259237,"avatar_url":"https://avatars0.githubusercontent.com/u/2259237?v=4","gravatar_id":"","url":"https://api.github.com/users/AndreyAkinshin","html_url":"https://github.com/AndreyAkinshin","followers_url":"https://api.github.com/users/AndreyAkinshin/followers","following_url":"https://api.github.com/users/AndreyAkinshin/following{/other_user}","gists_url":"https://api.github.com/users/AndreyAkinshin/gists{/gist_id}","starred_url":"https://api.github.com/users/AndreyAkinshin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AndreyAkinshin/subscriptions","organizations_url":"https://api.github.com/users/AndreyAkinshin/orgs","repos_url":"https://api.github.com/users/AndreyAkinshin/repos","events_url":"https://api.github.com/users/AndreyAkinshin/events{/privacy}","received_events_url":"https://api.github.com/users/AndreyAkinshin/received_events","type":"User","site_admin":false},"created_at":"2017-10-11T15:30:54Z","updated_at":"2017-10-11T15:30:54Z","author_association":"MEMBER","body":"@pakrym, than you can use a special job for your CI (see http://benchmarkdotnet.org/Configs/Jobs.htm).\r\nE.g. you can mark your benchmark with `DryJob` or `ShortRunJob` (see https://github.com/dotnet/BenchmarkDotNet/blob/1670ca349c303b83c373855705ea4f4679f99ad0/src/BenchmarkDotNet.Core/Jobs/Job.cs#L24), these kinds of jobs are designed for a quick validation that everything is fine with benchmarks (without actual high accuracy measurements).\r\n"}}
{"actor":{"display_login":"AndreyAkinshin"}}