I have a 8GB RPi4 (Raspberry Pi OS) running OMV to use it as a NAS. Is it possible to use the same RPi4 as a Nextcloud server?
There are two basic factors to consider, the minimum memory and processing power required. By "the minimum" I mean enough to start the software you want to use with a normative configuration.
Keep in mind that given enough demand, any and all servers can be overwhelmed. It does not matter how powerful the system (or conglomerate thereof) is, there is going to be a point wherein it maxes out in terms of dealing with requests. A usual bottleneck for a server is network bandwidth and this is likely to run out before RAM or CPU time.
Since Pi models with only one gig of RAM can run both pieces of software here, we can assume that is not a problem -- although again, the amount of memory required depends on the workload.
The same really applies to processing power -- if this is just you/your household, those apps will probably be idle much of the time. How fast they can do what you want when required is again likely to hit the network bandwidth bottleneck first, so processing power is unlikely to be a problem -- ie., the CPU can keep up with the network interface. Still, the amount of processing power required depends on the workload.