This is already the correct answer IMO, and what I am suggesting will not probably not make much difference, but it might be useful to know.
If all you want to do is run the browser, you don't have to run a desktop environment as well. Create a file which looks like this as
If .xinitrc already exists, move it temporarily or comment out anything else. Now,
startx (obviously, you should not be in it already -- do this from the console without the GUI running). Voila, you have just the browser, no desktop.
That saves a wee bit of memory, although the browser is by far the elephant in the room and the Xorg server itself (that's running) is bigger than a basic lxde (which now isn't running). If you have so much loaded into RAM that you are using swap, that will impact performance. The above midori + bare X uses < 100 MB resident according to
total used free shared buffers cached
Mem: 448708 242604 206104 0 82660 105156
-/+ buffers/cache: 54788 393920
Swap: 102396 0 102396
448708 - 393920 = 54788 / 1024 = 53.5 MB
That's with 4 tabs open. Again, if you look at these and see your RAM is near full, that's a performance issue. Note that it is normal to use a bit of swap even if the ram is not full, so don't worry about that -- that swapped stuff is low priority.
Something further to think about, performance wise, is the significance of buffers and cache. I did not include these in the total, and notice it is actually more than the committed memory (about twice as much). That is normal. If you fill the memory up with committed stuff, the system will just use less cache and/or transfer it to swap. Either way, that is going to be a performance degradation because the cache is important (it's just not vital or immutable size wise, so not part of the committed mem stat).
In other words, optimally you want your committed ram to be no more than 75% of what's available on the pi and perhaps less than that. If you use LXDE and start opening other stuff, you may quickly start to approach that.