I know this is pretty old, but I just had this same problem. The issue was that my ground from the board was incorrectly wired. With the USB port connected it acted as a ground, but as soon as it disconnected there was no longer a ground and the board crashed.