The following thread on the RPi forums discusses some differences between OpenELEC+Kodi and Raspbian+Kodi.
According to that discussion, a key difference is the memory split in Raspbian. Essentially, this determines how much of your RAM is available for running programs and how much is used by the GPU. If you allocate more RAM to the GPU, your Pi can perform more intensive graphics processing.
I suspect (but am not 100% sure) that to get Raspbian+Kodi working more effectively you'd have to adjust this memory split in order for the GPU to run Kodi at a decent speed. Although you'd probably be better off just running OpenELEC from one SD card and installing Raspbian on a second and switch between the two when you want to watch media or play with an OS respectively.