I see mainly two possibilities for your needs. You can setup a debian repository. Then you only need an initial SD Card installation that can be updated with apt from your own repository. How to do it you may have a look at Debian Repository Setup.
You can setup a net booting infrastructure. This is more flexible than an repository because you can load ...
There is a good whitepaper here which describes in detail the differences between Yocto and Debian(Raspbian is a flavor of Debian).
From the whitepaper, the gist is:
Debian is good for fast trials, proof of concept and desktop like
environments on hardware already supported by Debian with large
memories and limited requirements for maintenance, ...