A clear and simple way to distinguish "system files" from "user data", the latter including custom installed software (vs. software installed from the distro repository via the package manager) is to consider most of the root filesystem to be the former, and the latter specific directories such as
You can then use a second partition for the user files,1 allowing you to mount the primary partition (with the basic root fs) read-only, and symlink the user directories to a location in the (mounted) second parition. For example, if the second partition is
/dev/mmcblk0p3, you can mount this at boot time via
/dev/mmblk0p3 /mnt/user ext4 defaults 0 2
You may want to use a UUID instead of the device node. It is probably best that you use
ext4 or some other POSIX compatible fs type (ie., not vfat, etc) since parts of the root filesystem proper will be probably be located there.
You then create parallel directories on the second partition for the stuff you want to move to it, by moving them to it. In other words, make a directory
/mnt/user/usr (while the second partition is mounted there!) and move
/mnt/user/usr/local. You now have no
/usr/local, which hopefully there's nothing critical there that now breaks (there isn't anything by default, so you would be aware of this), but you then create a symlink:
ln -s /mnt/user/usr/local /usr/local
You can do the same with
/home or subdirectories thereof, and even subdirecories of
/etc/ if you want specific stuff there writable, same with
/var/log and so on.2
A complication is how much of your application software is installed from the distro, which as a rule of thumb should be as much of it as possible. Don't go build a mongoDB in
/usr/local just to include it there. Use the distro packages and symlink the configuration, and/or configure the configuration to use runtime directories in
/usr/local (conventionally there's also an
etc there, it is empty in a base image).
This will take some experimenting/playtesting to see what stuff gripes when the root fs is mounted read-only. You probably also want to occasionally remount it read-write and do a update via the package manager. This will not tinker with stuff in
I'm going to refer to the root fs partition, on Raspbian
/dev/mmcblk0p2 as the "first partition" and the extra one as the second, even though the real first partition is the one default mounted on
/boot (a method parallel to what I'm recommending for your user stuff). Note in your scenario you might as well leave that boot partition unmounted (it is not required to be).
Note that you'll still have a writable
tmpfs partition mounted on
/run or subdirs thereof, can't remember exactly which way Raspbian does it, but this should not interfere (and you want those writable!).