The number of server processes it is possible to have running in parallel is not a meaningful constraint, unless some of those processes require unusual amounts of memory even when doing nothing, which generally speaking server processes do not.
The constraint is how much those servers can do in parallel; you could overwhelm the Pi with only one given enough load.
So you have to determine what the typical load is going to be, find out by experimentation what this requires in terms of memory, processor use, and bandwidth. The last one is probably what will add up the fastest, ie., you won't be maxing out your RAM or processor because the servers will be limited by how much I/O they can do.
If this is for more or less personal use (<5 users), most likely you can run a quite a lot of things.
A while ago I wrote this for assessing the behavior of long running processes, primarily memory wise. It does not track I/O but there are such tools around which should be easy to find and use, eg.:
These popped up by searching "linux per process network usage". The brand of hardware isn't really relevant here so including "Raspberry Pi" in your searches will only limit then usually in odd ways.
complications like: PiHole wants to install lighttpd, but I have nginx installed already
This is sort of inevitable with port 80. Port conflicts are usually easily solved by just configuring something to use an arbitrarily different port. Eg., many people do not use port 22 for ssh because of all the bogus traffic this attracts (not an issue if you're just running it inside a LAN). All it means is anyone who wants to use ssh has to specify a different port than normal, something that can done via a command line switch or in a client configuration.
In the case of port 80, commonly used alternatives are 8080 and 8181. It doesn't matter much if this is just for in house access -- you can use any port and include it in the url after a colon: http://raspberrypi.local:1234. You could instead use https exclusively, which defaults to port 443. You will then need an X509 certificate; a "self-signed" one which should be fine for in house stuff can be generated on the pi with openssl or gnutls.
That alternate port approach seems like it could apply in the context of your nginx use if it is more awkward to set pihole to a different port. It may also be possible to integrate pihole with nginx. If not, you should be able to proxy it with nginx depending on what url paths are in use.
Questions about nginx use are more appropriate to our larger sibling sites, Unix & Linux and ServerFault, or possibly Stack Overflow, as the brand of hardware is again irrelevant and these are much larger communities.
