There are several approaches to this problem (which are not Pi specific actually).
One is "fat binaries" where a binary includes code segments for multiple cpus, and the start up execution environment activates the best one.
Another is to generate system binaries using compiler options that limit the executable to the lowest common denominator set of instructions that work on all cpus in the family. This may limit performance, as newer instructions might allow faster execution. This sometimes can be mitigated by isolating machine specific high performance code in a shared library, and activating the most appropriate version of the shared library at boot.
Another approach is if there are only a few specialized instructions not available on older cpus, install a trap for illegal instructions, and when it is triggered, emulate the missing instruction or back patch a subroutine call in its place.
Sometimes the differences between systems are more along the lines of different devices available, in which case the kernel can probe what is available at startup and either compensate for missing hardware or just not make it available.
The pi (and linux in general) uses several of these approaches at once.