I encounter a very annoying problem with My Raspberry Pi which is rebooting whenever I plug into it a Wireless USB adapter or an ethernet adapter.
As you can read here, 700mA is barely enough to run the B Model - I'm using a 1.2A Power supply and this still happens sometimes (using Model B Rev. 2) , I'd suggest getting a powered hub, then you won't have to worry about what you plug in/out.
edit: the polyfuses you talked about in the comment have been removed in Model Revision 2. Are you using a Rev 1 board?
edit 2: Oh, and overclocking the Pi will use more power - so if you did that, try going back to the standard level.