Raspberry Pi's core micro (BCM2835) doesn't have any audio outputs per se. Yes, it has the HDMI audio output, but the stereo 3.5mm jack outputs are forged using a pair of PWM generators rather than audio quality DACs.
DAC is "Digital to Analog Converter". There are at least 3 key ways to output an analog voltage from a micro. By comparison, PWM (Pulse Width Modulation) is fairly inexpensive in terms of hardware. It operates on a principle that you output a square wave, but you vary the length of time the output is "high" vs. the length of time it's "low". The average output, then, is the voltage. Different levels of voltage are required to push and pull the speaker core to different positions with different speeds to create different audio amplitudes and frequencies.
The problem with using a PWM as an audio DAC is that it's switching on and off quite rapidly and this, in itself, creates a frequency. Not something you want when you're outputting sound. So, you try to smooth (average) it by grounding it through a resistor and capacitor in parallel. Even then, the greater the ratio between "mark" (on time) and "space" (off time), the less "smooth" this average can be. The more "off" time there is (i.e. the lower the volume) the less smooth the average.
The RC network values need to be optimised for some selected range. My finding is that the RPI's audio tends to work quite well at levels of 75% and above, but is otherwise abysmal.
I'm sure there are electronics buffs who would dispute this interpretation, but this is my attempt to pitch what I know to make the answer as useful as possible to as broad an audience as possible.
My advice? Buy a $4 USB audio DAC off your favourite internet auction site. They're very well supported with pre-installed kernel options.