I want to modify the distro according to my needs. [...] What about
the source for the actual distro? Is it all there? Am I insane?
The distro has sources, but these are either the same as or slightly patched versions of the original sources -- a distro does not really contain any original software per se (discounting the package manager), it is just an original configuration of such.
So there are the original sources and the probably mostly very similiar .src packages that are available via the package manager. I have never heard of someone rebuilding a particular distro from source, as this would be somewhat pointless, unless it is for use on a platform for which those sources have not been previously compiled (such as raspbian is to debian).
Bluntly then, it does sound a little insane. I would not consider compiling everything over again from source a very direct route toward "modify[ing] the distro according to my needs". To a certain extent, compile time options are useful, but I think only in a small minority of cases (for which you can source build without having to do the whole rest of the system).
The kernel is a little more straightforward, although unless there are specific things that you know you want from a kernel, rolling your own is a largely academic exercise (it won't make any real difference). I have cross compiled the raspbian 3.6.11+ kernel following the elinux guide using a crosstool-ng toolchain and it works, although there is a little bit of weirdness that followed from converting the 3.2.27+ /proc/config.gz to 3.6.11 -- eg, the ipt_LOG module was not built, which many people probably use in their iptables rules..that took a bit of tracking down.
I think your best is to first decide specifically what "your needs" are and then get opinions about how to address those specifics, rather than starting off down the "start from scratch" path. Customizing an install doesn't generally require a lot of re-compiling, whereas building one all over again will take you months (if ever) to sort out and probably end with you realizing the same thing (it was mostly a waste of time).