I assume the 5V you are supplying to the ADC is the reference voltage. As you might guess, any voltage level that serves as a "reference" should be reliable within a tight tolerance. Consequently, you want a stable voltage and you cannot expect that when powering the unregulated 5V rail from batteries that lose power over time.
If you were supplying the Pi from a conventional micro-USB power adapter, and any other loads were modest and not varying to a great extent, then you may have a stable 5V power to use as a reference.
If you need to use batteries (e.g., for mobility), you might try testing the operation of your ADC under varying voltages to determine the range where it works correctly. Then implement some form of power monitoring to alert you when the batteries need to be changed.
Another option, which is kludgy but might be acceptable for a non-commercial DIY project, is to connect a step-up voltage converter to the 3.3V rail that outputs 5V. The voltage on the 3.3V rail is regulated and very stable, so the output 5V will be stable, as well, even as your input battery power falls below 5V. Assuming the ADC requires minimal current from the reference voltage connection, the 3.3V rail & converter should be able to supply it without any problem. This is a creative solution (offered in the spirit of brainstorming) that attempts to work around the constraints and deficiencies in the power architecture of the Pi. You would have to test it as a POC to assess if it works for your configuration.