Recently, I was trying to install OpenCV with TBB support on raspbian. I wasn't able to do so. On raspberrypi.org forum, I read a comment that for TBB support, you need a real armhf OS not a raspbian. I ended up using ubuntu 14.04 on rpi 2 to use OpenCV with TBB. But I was wondering what's the major difference between those?
Either you misread the comment or the comment was miswritten. Current versions of Raspbian are armhf and have been for some time. There is also some potential confusion here because Debian and other major GNU/Linux distros use
armhf to refer to ARMv7+ versions, which will only work on the Pi 2/3, whereas Raspbian uses
armhf to refer to itself, which is compatible with some (probably, most existing) ARMv6 systems including all models of Pi, but not those without hardware floating point support (which all models of Pi have)
The "hf" is for hard-float and refers to the fact that the binaries have been compiled for an arm architecture with hardware support for floating-point arithmetic, something you probably know is fundamental to a lot of software. Non-hf would mean the kernel and native runtime library do software operations instead (needed on hardware which doesn't provide such support). This is obviously not good for performance.
As mentioned, there is more than one flavour of "arm". The A/B/+/0 pis are ARMv6 (more specifically, the "ARM1176JZ(F)-S" implementation, which includes hard-float capability); the 2 is ARMv7. ARMv7 is backward compatible with ARMv6 and therefore the same Raspbian userland can be used -- the only thing they changed for the 2 is the kernel. However, stuff compiled for ARMv6 will not include the improvements available in ARMv7, and therefore it may in theory not perform as well (although I have not seen any evidence of this, and my own experience running an ARMv7 distro on the Pi 2 leads me to believe the difference in practice is not significant).
There are now various ARMv7 distros which target the Pi 2 and 3 specifically.
The more recent Pi 3 is 64-bit ARMv8, but since this is again backward compatible the same old Raspbian works again. In this case, the same ARMv7 kernel as used on the Pi 2 is also viable, so there is no Pi 3 kernel; some hardware configuration difference are taken care of with a separate device tree overlay.