{"action":"closed","issue":{"url":"https://api.github.com/repos/nunit/nunit-console/issues/225","repository_url":"https://api.github.com/repos/nunit/nunit-console","labels_url":"https://api.github.com/repos/nunit/nunit-console/issues/225/labels{/name}","comments_url":"https://api.github.com/repos/nunit/nunit-console/issues/225/comments","events_url":"https://api.github.com/repos/nunit/nunit-console/issues/225/events","html_url":"https://github.com/nunit/nunit-console/issues/225","id":229841754,"number":225,"title":"SocketException thrown by nunit3-console.exe --explore option","user":{"login":"jnm2","id":8040367,"avatar_url":"https://avatars1.githubusercontent.com/u/8040367?v=3","gravatar_id":"","url":"https://api.github.com/users/jnm2","html_url":"https://github.com/jnm2","followers_url":"https://api.github.com/users/jnm2/followers","following_url":"https://api.github.com/users/jnm2/following{/other_user}","gists_url":"https://api.github.com/users/jnm2/gists{/gist_id}","starred_url":"https://api.github.com/users/jnm2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jnm2/subscriptions","organizations_url":"https://api.github.com/users/jnm2/orgs","repos_url":"https://api.github.com/users/jnm2/repos","events_url":"https://api.github.com/users/jnm2/events{/privacy}","received_events_url":"https://api.github.com/users/jnm2/received_events","type":"User","site_admin":false},"labels":[{"id":425067002,"url":"https://api.github.com/repos/nunit/nunit-console/labels/is:bug","name":"is:bug","color":"ee0701","default":false},{"id":425450115,"url":"https://api.github.com/repos/nunit/nunit-console/labels/pri:high","name":"pri:high","color":"d93f0b","default":false}],"state":"closed","locked":false,"assignee":{"login":"jnm2","id":8040367,"avatar_url":"https://avatars1.githubusercontent.com/u/8040367?v=3","gravatar_id":"","url":"https://api.github.com/users/jnm2","html_url":"https://github.com/jnm2","followers_url":"https://api.github.com/users/jnm2/followers","following_url":"https://api.github.com/users/jnm2/following{/other_user}","gists_url":"https://api.github.com/users/jnm2/gists{/gist_id}","starred_url":"https://api.github.com/users/jnm2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jnm2/subscriptions","organizations_url":"https://api.github.com/users/jnm2/orgs","repos_url":"https://api.github.com/users/jnm2/repos","events_url":"https://api.github.com/users/jnm2/events{/privacy}","received_events_url":"https://api.github.com/users/jnm2/received_events","type":"User","site_admin":false},"assignees":[{"login":"jnm2","id":8040367,"avatar_url":"https://avatars1.githubusercontent.com/u/8040367?v=3","gravatar_id":"","url":"https://api.github.com/users/jnm2","html_url":"https://github.com/jnm2","followers_url":"https://api.github.com/users/jnm2/followers","following_url":"https://api.github.com/users/jnm2/following{/other_user}","gists_url":"https://api.github.com/users/jnm2/gists{/gist_id}","starred_url":"https://api.github.com/users/jnm2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jnm2/subscriptions","organizations_url":"https://api.github.com/users/jnm2/orgs","repos_url":"https://api.github.com/users/jnm2/repos","events_url":"https://api.github.com/users/jnm2/events{/privacy}","received_events_url":"https://api.github.com/users/jnm2/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2017-05-19T00:46:09Z","updated_at":"2017-05-25T23:23:13Z","closed_at":"2017-05-25T23:23:13Z","body":"@JackUkleja commented on [Wed Oct 12 2016](https://github.com/nunit/nunit/issues/1834)\n\nI am running NUnit.ConsoleRunner.3.5.0\\tools\\nunit3-console.exe with the --explore option against one of my test assemblies and I am getting the following stack trace:\n\n> System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host\n> \n> Server stack trace:\n>    at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)\n>    at System.Runtime.Remoting.Channels.SocketStream.Read(Byte[] buffer, Int32 offset, Int32 size)\n>    at System.Runtime.Remoting.Channels.SocketHandler.ReadFromSocket(Byte[] buffer, Int32 offset, Int32 count)\n>    at System.Runtime.Remoting.Channels.SocketHandler.Read(Byte[] buffer, Int32 offset, Int32 count)\n>    at System.Runtime.Remoting.Channels.SocketHandler.ReadAndMatchFourBytes(Byte[] buffer)\n>    at System.Runtime.Remoting.Channels.Tcp.TcpSocketHandler.ReadAndMatchPreamble()\n>    at System.Runtime.Remoting.Channels.Tcp.TcpSocketHandler.ReadVersionAndOperation(UInt16& operation)\n>    at System.Runtime.Remoting.Channels.Tcp.TcpClientSocketHandler.ReadHeaders()\n>    at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)\n>    at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)\n> \n> Exception rethrown at [0]:\n>    at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)\n>    at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)\n>    at NUnit.Engine.ITestAgent.Stop()\n>    at NUnit.Engine.Runners.ProcessRunner.Dispose(Boolean disposing)\n>    at NUnit.Engine.Runners.AbstractTestRunner.Dispose()\n>    at NUnit.Engine.Runners.MasterTestRunner.Dispose(Boolean disposing)\n>    at NUnit.Engine.Runners.MasterTestRunner.Dispose()\n>    at NUnit.ConsoleRunner.ConsoleRunner.ExploreTests(TestPackage package, TestFilter filter)\n>    at NUnit.ConsoleRunner.Program.Main(String[] args)\n\nAny clues what is going on? The tests in this assembly run just fine, but the explore options seem to kill the runner.\n\n\n---\n\n@ChrisMaddock commented on [Wed Oct 12 2016](https://github.com/nunit/nunit/issues/1834#issuecomment-253158254)\n\n@JackUkleja - What error do you see if you add the `--inprocess` option?\n\n\n---\n\n@JackUkleja commented on [Wed Oct 12 2016](https://github.com/nunit/nunit/issues/1834#issuecomment-253377003)\n\nIf I try `--inprocess` I get:\n\n> NUnit.Engine.NUnitEngineException: Cannot run tests in process - a 32 bit process is required.\n\nSo obviously my test DLL is 32 bit and the nunit-console.exe is running as 64 bit due to it being 64 bit OS.\n\nIf I use Corflags to force nunit-console.exe to run as 32 bit then the `--explore` option appears to work just fine.\n\nSo it seems I am unable to replicate the original exception when doing `--inprocess`.\n\n\n---\n\n@CharliePoole commented on [Wed Oct 12 2016](https://github.com/nunit/nunit/issues/1834#issuecomment-253391478)\n\nIt's useful info nonetheless. We should see if we can replicate this.\n\n\n---\n\n@ChrisMaddock commented on [Thu Oct 13 2016](https://github.com/nunit/nunit/issues/1834#issuecomment-253492517)\n\nHmm...I can't reproduce this with a simple x86 dll. \n\n@JackUkleja - What's your entire command line, is there anything else involved? Could you try setting up a minimal example, with a 1 test assembly you'd be able to share? \n\n\n---\n\n@JackUkleja commented on [Mon Oct 24 2016](https://github.com/nunit/nunit/issues/1834#issuecomment-255904943)\n\nOne more clue about this...\n\nI was just trying to replicate this issue with the following command:\n\n> \"NUnit\\NUnit.ConsoleRunner.3.5.0\\tools\\nunit3-console.exe --explore Artefacts\\Debug\\Tests\\IntegrationTests.dll\"\n\nand much to my surprise it worked without any errors. I got a list back of the contained tests, as expected! So I tried a few of my other test assemblies - they all worked too. Finally I tried my IntegrationTests.dll once more, exactly the same command line,...but this time it failed with the aforementioned `SocketException`\n\n\n---\n\n@JackUkleja commented on [Mon Oct 24 2016](https://github.com/nunit/nunit/issues/1834#issuecomment-255906691)\n\nNothing I can do now seems to let me run this command successfully again. I've rebuilt the assemblies, restarted the console, looked for any \"remoting\" or \"nunit\" processes that maybe could be killed or restarted (nothing). Is something persisting in the remoting infrastructure that is breaking things - something that is not being cleaned up? I'll try log off/on to see if that gets it working again....\n\n\n---\n\n@JackUkleja commented on [Mon Oct 24 2016](https://github.com/nunit/nunit/issues/1834#issuecomment-255907025)\n\nOh and this only seems to be happening with my IntegrationTests.dll so there is clearly some strange specific interaction between that dll and nunit remoting.\n\nUpdate: The command just randomly succeeded without me changing anything specifically. I just ran it a bunch of times and about once in 50 attempts it worked again...\n\n\n---\n\n@rprouse commented on [Thu Dec 01 2016](https://github.com/nunit/nunit/issues/1834#issuecomment-264290849)\n\n@JackUkleja do you have any updates on this? I suspect that it is something in your tests and would like to close this issue.\n\n---\n\n@tfabris commented on [Wed Mar 22 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-288573833)\n\nOur company is getting this exact same error with this exact same stack trace. It is rare and intermittent for us. We have unit tests that run several thousand test cases on several different code branches several times per day, And we're hitting this perhaps twice per day overall. So it's quite rare for us. But we do hit it. So rprouse, please don't close the issue just yet.\r\n\r\nWe are running Nunit version 3.6.1 by the way.\r\n\r\n\n\n---\n\n@tfabris commented on [Wed Mar 22 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-288575192)\n\nTo clarify: When it happens, it's random as to which test case happens to be running when it hits. So we don't think it's related to something specific in any of our tests. 99% of the time the tests run just fine and they all pass, it's only on the rare occasion that we hit this error, and when we do, it's randomly somewhere in the test list, not the same place every time. (But when the crash occurs, it's always the exact same stack trace shown at the top of this thread.)\r\n\r\n\n\n---\n\n@rwencel commented on [Thu Mar 23 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-288833039)\n\nWe have the same problem, same description, though we don't use the explore option, we just run the tests in parallel.  100% of tests succeed, but at the end of the run we encounter this error.  It's sporadic.  One theory is that it's when our build server is under stress (multiple builds running at once), but I haven't confirmed that.\r\n\r\nI'm trying a build with the workaround mentioned by @eberlid in the issue below to see if it fixes.  I'm not sure it's a proper fix b/c it just ignores the exception, but it seems harmless and I'm not sure what the root cause is.\r\n\r\n[NUnit Console Runner Issue #171](https://github.com/nunit/nunit-console/issues/171)\n\n---\n\n@tfabris commented on [Thu Mar 30 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-290565615)\n\nThis issue continues to occur for us intermittently when using Nunit 3.6.1 to run our unit test and integration test suites on Team City.\r\n\r\nBecause it's so intermittent, we don't have a way to reliably reproduce the issue locally on our local machines. Is there any forensic information we can obtain from Team City that would be helpful?\r\n\r\nIt would be nice to be rid of these \"exit code -100\" socket exception errors, since they block production when they occur.\n\n---\n\n@ChrisMaddock commented on [Fri Mar 31 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-290648547)\n\nI also found one of our assemblies which hits this yesterday - but couldn't for the life of me manage to debug it. 😞\r\n\r\nIt feels like a race condition - as I can't reproduce it when running through visual studio or in debug mode. Logging makes it harder to come by - I have some logs, but they're not useful.\r\n\r\nThe only difference I can really think about in this assembly which is erroring is that it's far more dependent on testcasesources than some of our other assemblies. At some point, I'll try and test out that theory with some slow testcasesources.\n\n---\n\n@jnm2 commented on [Fri Mar 31 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-290694988)\n\nI've hit this in debugging a fair number of times a few months ago but none recently.\n\n---\n\n@ChrisMaddock commented on [Fri Mar 31 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-290703819)\n\nFor what it's worth - this is what the logs look like up until the crash, with `--trace=Debug`. I was running a single assembly, with just the `--explore` option.\r\n\r\nEngine\r\n```\r\n13:50:38.052 Debug [ 1] TestAgency: Waiting for agent {7c42103a-6371-4f5f-ba31-df7e7016a6a6} to register\r\n13:50:38.253 Debug [ 1] TestAgency: Returning new agent {7c42103a-6371-4f5f-ba31-df7e7016a6a6}\r\n13:50:41.227 Info  [ 1] ProcessRunner: Unloading \r\n13:50:41.313 Debug [ 1] ProcessRunner: Stopping remote agent\r\n13:50:41.321 Error [ 1] ProcessRunner: Failed to stop the remote agent. An existing connection was forcibly closed by the remote host\r\n(log continues - stops all services as expected)\r\n```\r\n\r\nAgent\r\n```\r\n13:50:38.294 Info  [ 5] NUnitFrameworkDriver: Loading Tests.exe - see separate log file\r\n13:50:39.956 Info  [ 5] NUnitFrameworkDriver: Loaded Tests.exe\r\n13:50:39.957 Info  [ 5] NUnitFrameworkDriver: Exploring Tests.exe - see separate log file\r\n13:50:41.315 Info  [ 5] RemoteTestAgent: Stopping\r\n(End of agent log)\r\n```\n\n---\n\n@tfabris commented on [Mon Apr 03 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-291224042)\n\n@rprouse and @CharliePoole , do you guys have any idea about where this is coming from? It only started occurring when we upgraded from NUnit 3.4.1 to NUnit 3.6.1, so it is definitely a new regression which occurred somewhere in between those two builds.\n\n---\n\n@CharliePoole commented on [Mon Apr 03 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-291229247)\n\n@ChrisMaddock Didn't you confirm this already?\n\n---\n\n@ChrisMaddock commented on [Mon Apr 03 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-291239592)\n\n@CharliePoole - yes, although only as in so much as I can see it happening, I've not got much idea what's wrong!\r\n\r\nI was hoping to take some of the characteristics of the assembly I see it happen with a lot, and create a repro I can share. Haven't managed to get back to that yet.\r\n\r\nUpdate: Nope...not what I thought unfortunately...\n\n---\n\n@eberlid commented on [Mon Apr 03 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-291276858)\n\nWe got the same symptoms when we began to start multiple Gallio test\r\nrunners in parallel using MbUnit test framework. Before that we did not\r\nstart multiple runners in parallel and we did not get this issue.\r\n\r\nProbably both runners have something in common?\r\n\r\nSee https://github.com/nunit/nunit-console/issues/31#issuecomment-267059306\r\n\n\n---\n\n@CharliePoole commented on [Mon Apr 03 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-291280999)\n\n@eberlid Insofar as I understand this issue is about running a single copy of the console runner using --explore option. \r\n\r\nI can envision all sorts of problems that may come up running multiple copies of the runner at the same time and we should actually have another issue for that. Each instance of the engine starts up a server to which the agents launched by that instance report. So with n consoles and n agents running, there's lots of room for unexpected stuff to happen, especially as this was not a use case we originally had in mind.\n\n---\n\n@tfabris commented on [Mon Apr 03 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-291281883)\n\n@CharliePoole, our repros are all running a single copy of the console runner on any given computer as far as I know. We're launching these tests from within a single build step in Team City, and we're not yet taking advantage of the parallelization features built into Nunit 3 yet.\r\n\r\nHowever, we do have a fleet of multiple test agent computers on the network. So more than one testrun might be executing simultaneously on completely different computers. I'm assuming that by \"single copy of the console runner\", you mean, a single copy on that particular computer.\n\n---\n\n@CharliePoole commented on [Mon Apr 03 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-291282186)\n\n@tfabris Yes, that's what I meant - i misunderstood what you were saying.\n\n---\n\n@tfabris commented on [Mon Apr 03 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-291282750)\n\nWell, maybe our situation is different than @eberlid. I don't know how MbUnit runs things.\n\n---\n\n@CharliePoole commented on [Mon Apr 03 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-291283273)\n\n@tfabris Sorry to confuse... my comment was meant for @eberlid\n\n---\n\n@JackUkleja commented on [Mon Apr 03 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-291333032)\n\n@CharliePoole  OP here. Just FYI my gut feeling is that --explore option is probably not salient to this bug, but I have not experienced the issue since originally posting and have not personally seen it during normal test runs.\r\n\r\nIt is interesting this thread had a spate of repros in the last couple of weeks. Does this coincide with a new release? Or perhaps this issue is showing up more in organic search? I'm just surprised at the sudden influx of repro after 3 or 4 months of silence.\n\n---\n\n@tfabris commented on [Mon Apr 03 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-291337419)\n\n@JackUkleja, that's probably my fault. The spate probably started when I came a-posting a couple of weeks ago. In my particular case, it was because we had been on NUnit 3.4.1 until then. We upgraded to NUnit 3.6.1 on our Team City server, and that's when suddenly we started seeing our Team City builds getting all these \"the process exited with code -100\" errors. Then I did some searching and came across this thread showing the exact same stack trace, and posted about it,, which woke things up again for this thread.\r\n\r\nWe run thousands of unit tests and integration tests every day, and we encounter this error a few times each day. So that should give you an idea of the frequency of the problem, though I don't have exact numbers.\r\n\r\nEven though that seems like a rare frequency for this crash, it has an interesting effect on us because the tests are a big part of our production chain. When the tests fail with this crash, it's brings our production grinding to a halt and we have to re-run everything from the top of the chain again. So we're keen on finding either a solution or at least a work-around.\n\n---\n\n@kditrj2d commented on [Mon Apr 17 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-294604822)\n\nHello,  I am another user who is encountering this exact same exception and stack-trace.  For me the issue has been happening for a while.  It was rare (once a day or so) when we were running NUnit 2.6 tests with the NUnit 3.6.1 console runner.  We have just upgraded our tests to NUnit 3.6.1 and the problem has become more serious, so much so that it is blocking our CI build.  We are currently running CI from within Team City.  The last test assembly reported before the crash runs perfectly happily on the command-line and from within VS2015 via the R# test runner.  I haven't tried running with --inprocess yet.  I don't believe that any of our tests are currently using any of the parallelism features available in NUnit 3.\n\n---\n\n@baluMallisetty commented on [Tue Apr 18 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-294886410)\n\nHello, Please check if there is any TCP connection being established along a NAT channel. .net has remoting issue with TCP over NAT. Use WCF if possible.\n\n---\n\n@tfabris commented on [Mon May 08 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-299928959)\n\n@CharliePoole any word on this issue? We continue to encounter it randomly among our thousands of unit and integration tests, when running Nunit 3.6.1.\r\n\r\nIt only seems to occur when we have the tests running on Team City agent servers. I don't think any of our developers has reported it happening when they run tests on their local computers. I'm not sure if that's due to the special command line parameters that Team City uses to launch the tests, or if it's merely a quantity thing (our Team City servers are where the tests get run most frequently).\n\n---\n\n@tom-dudley commented on [Mon May 08 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-299977363)\n\nWe've also been seeing the same exact message when running via Teamcity, using 3.6.0. The arguments being used are:\r\n\r\n> nunit3-console.exe foo.nunit --result=bar.nunit.xml --noheader --framework=net-4.0 --where \"cat != SomeTests\" --explore=baz_all.tests\r\n\r\nWe're currently avoiding the issue by using `--inprocess`\n\n---\n\n@tfabris commented on [Mon May 08 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-299985978)\n\n@tom-dudley - Thanks for the tip, I will experiment with adding \"--inprocess\" as a temporary workaround and see if it solves the problem for us.\r\n\r\nIf I understand it correctly, this prevents parallelism for test fixtures which support parallelism? So it would only be a useful workaround for folks who don't use the parallel features. Am I correct in that understanding?\n\n---\n\n@tfabris commented on [Mon May 08 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-299995391)\n\n@tom-dudley - I take that back. I'm not able to use \"--inprocess\" as a workaround because some of our test cases must be run with the --x86 parameter, and if I try to add \"--inprocess\" I get an error message saying that the two parameters are incompatible.\r\n\r\n@CharliePoole - Note: Possible help to narrow this down... At least one of the places where we were getting this socketException error is specifically in our 32-bit tests which were running with the --x86 parameter. I don't have data on whether all instances of the socketException that we encountered were \"--x86\" cases or not.\n\n---\n\n@tom-dudley commented on [Mon May 08 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300025825)\n\nI managed to strike lucky and find an assembly where this is mostly reproducible. Everything in the Dispose  looks to be working fine up to [calling _agent.Stop()](https://github.com/nunit/nunit-console/blob/master/src/NUnitEngine/nunit.engine/Runners/ProcessRunner.cs#L244) in `ProcessRunner`. Sticking in a call to the agent in the line before works, so the connection seems fine at this point. We then step over to the `RemoteTestAgent.Stop` code and execute [stopSignal.Set()](https://github.com/nunit/nunit-console/blob/master/src/NUnitEngine/nunit.engine/Agents/RemoteTestAgent.cs#L105). After that line executes we immediately hit error handling back in `ProcessRunner` and bubble it up. So I wonder if this `ManualResetEvent` is causing us to exit prematurely before we're actually done with the object. The termination logic for the agent is [here](https://github.com/nunit/nunit-console/blob/dc655ce241994f5a3664fe034b19d7be04035b64/src/NUnitEngine/nunit-agent/Program.cs#L181-L192).\r\n\r\nI can't see any obvious reason why this might fail, but someone else might spot something. If anyone has any other ideas for things for me to try with this assembly I'm all ears. I'll see about putting together a minimal one if the issues continues on.\r\n\r\nOne thing of note however was that if I run `nunit3-console` without the framework assembly in the same directory, my test assembly fails 99% of the time. But if I drop the framework assembly in the folder then everything works. Not sure what's going on here! (I'm checked out at 3.6.1 for this work)\n\n---\n\n@tfabris commented on [Mon May 08 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300035244)\n\n@tom-dudley and @CharliePoole - If it helps, I did some history digging and found that the socketException has occurred randomly on both our regular 64-bit tests as well as our 32-bit tests that were run with the --x86 command line parameter. So it's not related to that.\n\n---\n\n@rprouse commented on [Mon May 08 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300041288)\n\n@tom-dudley I also don't see anything obvious. Since you are able to fairly reliably reproduce this, what about wrapping the `stopSignal.Set()` and the `stopSignal.WaitOne(timeout)` in try/catch to see if it is one of these that is throwing and if so, see if you can get more info out of the exception. Maybe even do the same for the `while` loop in `WaitForStop()`?\r\n\r\nI am hoping that we can start narrowing down exactly what is throwing the exception that seems to be killing the agent.\n\n---\n\n@rprouse commented on [Mon May 08 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300043147)\n\n@CharliePoole and @ChrisMaddock is there any reason we haven't moved this to the NUnit Console repo? Is it only because it has the confirm label?\n\n---\n\n@rprouse commented on [Mon May 08 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300044473)\n\n@CharliePoole, just curious why we use a `TcpChannel` rather than an `IpcChannel`. I would think that an `IpcChannel` would be more reliable and faster since we never run agents on other machines. I don't think it would fix this issue, but I am curious about the history. Was it because we intended to run agents on other machines? If so, should we rethink that until it is needed?\n\n---\n\n@tom-dudley commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300085701)\n\nTurns out the `RemoteTestAgent` is exiting cleanly as far as it is concerned. It always hits `return 0` in `Main`.\r\n\r\nI believe the following code may reliably reproduce the error. I still don't understand why this causes the exception to be thrown though!\r\n\r\n```\r\npublic class Tests\r\n{\r\n    public static readonly string[] TestDirectory = { TestContext.CurrentContext.TestDirectory };\r\n\r\n    [Test]\r\n    public void MyTest([ValueSource(nameof(TestDirectory))] string directory)\r\n    {\r\n    }\r\n}\r\n```\n\n---\n\n@ChrisMaddock commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300096431)\n\nI also have an assembly which produces this fairly reliably with `--explore` - but I haven't been able to create anything reproducible. 🙁 Anything I've done to add logs, or try and catch the exception, seems to suppress the error - I guess by slowing the process down.\r\n\r\n> is there any reason we haven't moved this to the NUnit Console repo?\r\n\r\nI only haven't moved it as we haven't tracked down anything to actually fix yet. It's also likely a duplicate of some of the below - but I didn't think we could really confirm that, till we had something to test... 🙁 \r\n\r\nhttps://github.com/nunit/nunit-console/issues/171\r\nhttps://github.com/nunit/nunit/issues/2027\r\nhttps://github.com/nunit/nunit-console/issues/219\n\n---\n\n@jnm2 commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300201325)\n\nI can manage to get a socket exception once in a while by repeatedly running `.\\nunit3-console.exe mock-assembly.dll mock-assembly.dll mock-assembly.dll [...] --process=multiple` against 3.6.1:\r\n\r\n```\r\nUnhandled Exception: System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host\r\n\r\nServer stack trace:\r\n   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)\r\n   at System.Runtime.Remoting.Channels.SocketStream.Read(Byte[] buffer, Int32 offset, Int32 size)\r\n   at System.Runtime.Remoting.Channels.SocketHandler.ReadFromSocket(Byte[] buffer, Int32 offset, Int32 count)\r\n   at System.Runtime.Remoting.Channels.SocketHandler.Read(Byte[] buffer, Int32 offset, Int32 count)\r\n   at System.Runtime.Remoting.Channels.Tcp.TcpFixedLengthReadingStream.Read(Byte[] buffer, Int32 offset, Int32 count)\r\n   at System.IO.BinaryReader.ReadBytes(Int32 count)\r\n   at System.Runtime.Serialization.Formatters.Binary.SerializationHeaderRecord.Read(__BinaryParser input)\r\n   at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadSerializationHeaderRecord()\r\n   at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run()\r\n   at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)\r\n   at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)\r\n   at System.Runtime.Remoting.Channels.CoreChannel.DeserializeBinaryResponseMessage(Stream inputStream, IMethodCallMessage reqMsg, Boolean bStrictBinding)\r\n   at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)\r\n\r\nException rethrown at [0]:\r\n   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)\r\n   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)\r\n   at NUnit.Engine.ITestAgent.Stop()\r\n   at NUnit.Engine.Runners.ProcessRunner.Dispose(Boolean disposing) in C:\\Users\\Joseph\\Source\\Repos\\nunit-console\\src\\NUnitEngine\\nunit.engine\\Runners\\ProcessRunner.cs:line 258\r\n   at NUnit.Engine.Runners.AbstractTestRunner.Dispose() in C:\\Users\\Joseph\\Source\\Repos\\nunit-console\\src\\NUnitEngine\\nunit.engine\\Runners\\AbstractTestRunner.cs:line 225\r\n   at NUnit.Engine.Runners.TestExecutionTask.Execute() in C:\\Users\\Joseph\\Source\\Repos\\nunit-console\\src\\NUnitEngine\\nunit.engine\\Runners\\TestExecutionTask.cs:line 47\r\n   at NUnit.Engine.Runners.ParallelTaskWorkerPool.ProcessTasksProc() in C:\\Users\\Joseph\\Source\\Repos\\nunit-console\\src\\NUnitEngine\\nunit.engine\\Runners\\ParallelTaskWorkerPool.cs:line 81\r\n   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)\r\n   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)\r\n   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)\r\n   at System.Threading.ThreadHelper.ThreadStart()\r\n```\r\n\r\n\n\n---\n\n@jnm2 commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300201529)\n\nIf someone has a consistent repro, would be fun to play with that.\n\n---\n\n@tom-dudley commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300202451)\n\n@jnm2 The snippet I posted above is consistent for me. I just realised when I said it didn't show the error any more I was running the wrong assembly. Doh!\n\n---\n\n@jnm2 commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300210438)\n\n@tom-dudley  And is this without TeamCity?\n\n---\n\n@tom-dudley commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300211381)\n\nYep, this is creating a new project with that snippet, using nunit-console tagged at `3.6.1` and running it locally with `nunit3-console.exe .\\NUnitFailure\\bin\\Debug\\NUnitFailure.dll --explore`\n\n---\n\n@jnm2 commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300213951)\n\nMy debug and release builds at 3.6.1 don't repro this.\r\n\r\nAh, repro'd with https://github.com/nunit/nunit-console/releases/download/3.6.1/NUnit.ConsoleRunner.3.6.1.nupkg. 🎉 \n\n---\n\n@tom-dudley commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300215492)\n\nYes I repro'd with both that nuget package and with building `nunit-console` from source at the `3.6.1` tag. I've not tried the zip or msi on the Releases page of nunit-console.\n\n---\n\n@tom-dudley commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300226321)\n\nI have an assembly which contains a solitary reference to `nunit.framework` and it still gives the error. \n\n---\n\n@ChrisMaddock commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300226503)\n\nThis repros for me too. Awesome @tom-dudley, I haven't ever managed to break something out that did! 😄 \r\n\r\nMy assembly I regularly see this with doesn't use TestContext (It mostly pre-dates it!). It does however make extensive use of ValueSources.\n\n---\n\n@jnm2 commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300227847)\n\nAlso repros with:\r\n\r\n```c#\r\npublic class Tests\r\n{\r\n    public static readonly string[] TestDirectory = { \"\" };\r\n\r\n    static Tests()\r\n    {\r\n        _ = TestContext.CurrentContext;\r\n    }\r\n\r\n    [Test]\r\n    public void MyTest([ValueSource(nameof(TestDirectory))] string directory)\r\n    {\r\n    }\r\n}\r\n```\n\n---\n\n@baluMallisetty commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300227945)\n\nHello guys,\n\nI don't think I should in this conversations, I believe my email is ccied\nby accident, could you please loop me out?\n\nI am not aware of what Nunit is too! I think I am not the right person\nincluded in your email! 😂\n\nThanks\nBalu\n\nOn Tue, May 9, 2017 at 12:43 PM Chris Maddock <notifications@github.com>\nwrote:\n\n> This repros for me too. Awesome @tom-dudley\n> <https://github.com/tom-dudley>, I haven't ever managed to break\n> something out that did! 😄\n>\n> My assembly I regularly see this with doesn't use TestContext (It mostly\n> pre-dates it!). It does however make extensive use of ValueSources.\n>\n> —\n> You are receiving this because you commented.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/nunit/nunit/issues/1834#issuecomment-300226503>, or mute\n> the thread\n> <https://github.com/notifications/unsubscribe-auth/ALrPmS4rfmJ3KsajabZ34yhEOih4VZAKks5r4JebgaJpZM4KUc6I>\n> .\n>\n\n\n---\n\n@jnm2 commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300228420)\n\n@baluMallisetty We are not in control of that. You can visit this page (https://github.com/nunit/nunit/issues/1834) and click Unsubscribe in the sidebar for this thread.\r\n\n\n---\n\n@jnm2 commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300228647)\n\nAlso repros with:\r\n\r\n```c#\r\npublic class Tests\r\n{\r\n    public static readonly string[] TestDirectory = { \"\" };\r\n\r\n    static Tests()\r\n    {\r\n        _ = TestContext.CurrentContext;\r\n    }\r\n\r\n    [TestCaseSource(nameof(TestDirectory))]\r\n    public void MyTest(string directory)\r\n    {\r\n    }\r\n}\r\n```\n\n---\n\n@jnm2 commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300229316)\n\nMy guess is that the trigger is when NUnit needs to invoke a member in order to enumerate the tests and the class containing the member has a static constructor that accesses TestContext.CurrentContext.\n\n---\n\n@jnm2 commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300235239)\n\nIt's not specifically the constructor, it's just caused by the `TestContext.CurrentContext` being accessed when NUnit invokes a member in order to enumerate test cases.\r\n\r\n```c#\r\npublic class Tests\r\n{\r\n    public static string[] TestDirectory()\r\n    {\r\n        _ = TestContext.CurrentContext;\r\n        return new[] { \"\" };\r\n    }\r\n\r\n    [TestCaseSource(nameof(TestDirectory))]\r\n    public void MyTest(string directory)\r\n    {\r\n    }\r\n}\r\n```\r\n\r\nGotta take a break for a while.\n\n---\n\n@tom-dudley commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300236181)\n\nJust to remove all usages of strings:\r\n```\r\npublic class Tests\r\n{\r\n    public static readonly TestContext context;\r\n\r\n    static Tests()\r\n    {\r\n        context = TestContext.CurrentContext;\r\n    }\r\n\r\n    [TestCaseSource(nameof(context))]\r\n    public void MyTest(TestContext context)\r\n    {\r\n    }\r\n}\r\n```\n\n---\n\n@tom-dudley commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300269559)\n\nAfter a bit of cowboy-debugging:\r\n```\r\n<properties>\r\n    <property name=\"_SKIPREASON\" value=\"System.InvalidCastException : Unable to cast object of type 'NUnit.Framework.TestContext' to type 'System.Collections.IEnumerable'.\"/>\r\n    <property name=\"_PROVIDERSTACKTRACE\" value=\" at NUnit.Framework.TestCaseSourceAttribute.GetTestCaseSource(IMethodInfo method) at NUnit.Framework.TestCaseSourceAttribute.GetTestCasesFor(IMethodInfo method)\"/>\r\n</properties>\r\n```\n\n---\n\n@jnm2 commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300275232)\n\n@tom-dudley That's because you did `public static readonly TestContext context;` instead of `TestContext[]`. It should be an enumerable containing one item per test case.\n\n---\n\n@tom-dudley commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300282218)\n\n@jnm2 Good point! After stepping into the `TestCaseSourceAttribute` everything looked fine and I got some sensible XML back. Then I hit the `_agent.Stop()`, the agent gets to existing its `Main` and we get the exception.\n\n---\n\n@jnm2 commented on [Wed May 10 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300535011)\n\nAh, it's more complex.\r\nRight now I can consistently repro if and only if the test assembly has `nunit.framework.dll` in the same folder and `nunit3-console.exe` **does not** have `nunit.framework.dll` in the same folder.\n\n---\n\n@jnm2 commented on [Wed May 10 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300536216)\n\nIf this is accurate, we will not be able to do tests on this scenario without `build.cake` running `nunit3-console.exe` from a separate folder. I do think tests on this are important.\n\n---\n\n@jnm2 commented on [Wed May 10 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300537546)\n\nHey, I'm getting the repro to happen intermittently in a modified build.cake which copies `nunit3-console.exe` to its own folder before running mock-assembly tests. This is on a fresh debug build of nunit-console.\n\n---\n\n@tom-dudley commented on [Wed May 10 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300538702)\n\n@jnm2 Regarding the non-presense of `nunit.framework.dll`, I got exactly the same results as you - i.e. `nunit-console.exe` not having `nunit-framework.dll` in the same folder causing it to repro\n\n---\n\n@tfabris commented on [Wed May 10 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300555173)\n\nHm. Is it possible that the issue could be caused by a mismatched version of nunit-framework.dll?\r\n\r\nI notice that our tests, when run on Team City, which repro the issue only randomly/intermittently, seem to have the 3.4.1 version of nunit-framework.dll on the hard disk in the folder along with the test assembly. This must be a leftover in the build system that we haven't corrected yet. The tests get run with the 3.6.1 console runner, but there is a 3.4.1 version of nunit-framework in the folder with the tests.\r\n\r\nI'm going to try getting our build system to update to the 3.6.1 version of the framework.\n\n---\n\n@jnm2 commented on [Wed May 10 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300569850)\n\nI'm pretty sure the console runner acts differently depending whether the framework DLL is present, and perhaps it's version related, but it should not cause this exception no matter what.\n\n---\n\n@jnm2 commented on [Wed May 10 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300575342)\n\nIt turns out to be pretty tricky getting this under automated testing, but it's so worth it.\r\n\r\nThe test can't be done by an process that references or loads nunit.framework due to the Heisenburg quality of this bug, so the test has to be written in an assembly that does not use nunit.framework. Meaning, no testing framework.\r\n\r\nOr, if we're willing to forgo easy debugging, it can be done out of process. That's probably the least disruptive option.\n\n---\n\n@jnm2 commented on [Wed May 10 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300600527)\n\nHey guys, guess what? I have a reliable test at https://github.com/nunit/nunit-console/pull/223.\n\n---\n\n@tom-dudley commented on [Wed May 10 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300600835)\n\n- `nunit3-console` with no framework in the folder. Test linked against 3.6 - **Exception**\r\n- `nunit3-console` with framework 3.6 in the folder. Test linked against 3.6 - **No Exception**\r\n- `nunit3-console` with framework 3.7 in the folder. Test linked against 3.6 - **Exception**\r\n\r\n\r\n- `nunit3-console` with no framework in the folder. Test linked against 3.7 - **Exception**\r\n- `nunit3-console` with framework 3.6 in the folder. Test linked against 3.7 - **Exception**\r\n- `nunit3-console` with assembly 3.7 in the folder. Test linked against 3.7 - **No Exception**\r\n\r\nAnd by '3.7 in the folder', I just mean changing the assembly version number. All exceptions were repros of the original issue.\n\n---\n\n@tom-dudley commented on [Wed May 10 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300603222)\n\n@jnm2 The tests do a pretty good job at crashing my VS test runner, so looks like they're definitely reliable!\n\n---\n\n@jnm2 commented on [Wed May 10 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300605418)\n\nHave to take a break again, anyone is welcome to help find a fix if they so desire. I'll start looking at a fix when I'm back.\n\n---\n\n@jnm2 commented on [Wed May 10 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300605901)\n\n@tom-dudley Nice! That's valuable information that should probably be tested against too.\n\n---\n\n@tfabris commented on [Wed May 10 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300638563)\n\nThis is amazing work you are all doing. Awesome. In the meantime, my boss has updated our build system so that the framework and the console runner match at version 3.6.1, so we'll have additional data about whether it might be related to the version mismatch.\n\n---\n\n@CharliePoole commented on [Wed May 10 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300650422)\n\nI'm joining in late, so may be missing something. However...\r\n\r\nIt's absolutely certain that the framework you are  referencing has to be either in the same folder as your test assembly or somewhere else (like on the probing path) where it can be found. Consequently, there can't be another version of the assembly there, since all versions have the same file name. The framework is an assembly that you - not the runner - reference, so it's up to you to get it there.\r\n\r\nIn this context \"up to you\" includes any third party software you are using, like TeamCity or even VS. 😄\r\n\r\nOTOH, there is no requirement for the version of the runner to match that of the framework. The runner doesn't reference the framework. Of course, if you are running the nunit3-console runner, your framework had better be >= to 3.0 OR you had better have the V2 framework driver extension. That doesn't sound like your problem here, so I only mention it for completeness.ere can also be a loss of function if the runner is older than the framework - that is, the framework may have features that the runner doesn't understand - but things should still run.\r\n\r\n@jnm2 Re Runner acting differently if the framework isn't there... Yes... if the framework is not located with the test assembly that references it, we never find it to load and crash. That may not give the friendliest message and so should be checked.\r\n\r\nGuys... why is this an nunit bug rather than a console/engine bug?\n\n---\n\n@tfabris commented on [Mon May 15 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-301625984)\n\nI have confirmed that the issue still occurs on our build/CI system, intermittently, even when the framework version and the console runner version are both 3.6.1, and the framework DLL is present in the folder with the test assembly DLL.\n\n---\n\n@jnm2 commented on [Thu May 18 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-302578684)\n\n@CharliePoole \r\n> Guys... why is this an nunit bug rather than a console/engine bug?\r\n\r\nIt was never moved. @rprouse asked the same above.\n\n"}}
{"actor":{"display_login":"rprouse"}}