Not a complete answer, but I think it's something to do with the on-board chip.
I spent several days trying to fix this, trying everything I could find online, including trying multiple different OS's. Nothing worked.
I just tried some cheap $30 USB adapter that a friend bought on Amazon though, and it worked immediately. I'd save your time and just buy an adapter unless that isn't an option for some reason. Using the on-board chip also has the severe limitation that you obviously loose your wireless SSH connection as soon as you switch to monitor mode, meaning you have to do everything over a wired connection (unless you have it done via an automated script).
To address the accepted answer, assuming the OP has the same problem as me (it certainly sounds like it), this is not an issue related to memory or CPU usage. While the device is lagging, CPU usage and memory are near-zero; as reported by
airodump-ng seems to be the trigger, and the only resolution is to restart the device.