Turning ADB on or off when Android phone is in accessory mode removes it from accessory mode
===

I'm writing and application for the Nexus S that interfaces with my computer.

After I set the phone in accessory mode, any time I turn ADB on or off while it is in accessory mode, the phone leaves accessory mode and returns to regular USB mode, throwing my app off it's path.

Is this behavior to be expected? Can anything be done to keep the device in accessory mode AND turn ADB on or off?
I want it to go from Accessory Mode w/ ADB on mode to Accessory Mode w/ ADB off mode, and vice versa, but what's happening is that it's going from Accessory Mode w/ ADB on or off to Regular USB Mode w/ ADB off or on.

I am trying this in a linux VM, could that be the cause?

Also, I could swear I had it working last night and not sure what change caused this behavior to happen.