The subject of ADC error is both wide and deep. Engineers have built careers on studying, quantifying, measuring and resolving ADC error. If you "Google" terms such as adc error analysis, or adc error correction, you'll find enough highly technical and/or scholarly information to keep you occupied for quite some time - for example. But the bottom line is that power supplies can make a huge difference in ADC performance.
The description you gave of your PS sounds like a recipe for poor measurements. All switching power supplies will produce some noise, and HF oscillations "come with the territory" as the laws of physics dictate that the "off-then-on" nature of the supply will propagate to the output as oscillations. These oscillations are not necessarily problematic as long as they are not "too large"; "too large" being relative to the accuracy required. What is "too large" for your case may be determined by reading the data sheet for the ADC chip you're using; you should refer to that for your power supply specifications. One potential cure for taming the oscillations is a high quality, low ESR capacitor (tantalum, for example) that is placed as close as possible to your ADC circuitry.
Your comment about long-term drift ("over hours I see it slowly dropping") is what seems to be the strongest indicator that you have a sub-standard power supply. Again, some drift is normal and expected, but I get the idea that yours may be greater than that. I know of no easy way to correct drift in a power supply - except to replace it!
If your supply is one that you bought for your RPi, there is a decent chance it's not very good quality. And as mentioned, the only real solution here is to buy a better quality supply. For this, you may have to venture outside the "Raspberry Pi marketplace".
Hope this helps. Don't hesitate to ask additional questions, or post a Comment if something is unclear. Also, you may wish to consider adding a schematic to your question if you need more specific answers.
