I have a RPi4 now, have cooling sinks mounted on the processors and it runs pretty smooth, but sometimes I worry about the temperature development of my processor. At what level should I worry, and consider a fan?
You needn't worry at all if your only concern is temperature. All RPi models have a built-in closed-loop thermal control system that will prevent your RPi from overheating. This system operates by reducing the operating frequency and core voltage when the temperature reaches set limits. These frequency & voltage reductions are called throttling. Throttling reduces the power consumed in the SoC, which reduces heat/temperature - it also reduces performance of the system.
Understanding the operation of the RPi thermal management system might lead one to reconsider the question you have posed. You can see now that since the system manages temperature, performance is the variable one must reckon with. And so I say the answer to your question is this:
Consider a fan only when your RPi is being throttled to the point that its performance becomes insufficient to meet your needs.
Wrt the relationships between temperature and performance, there are numerous other details you may wish to explore. While the RPi's thermal management system is implemented in proprietary firmware, there are user-accessible configuration parameters available in the RPi that allow you to change some aspects of its performance. If you're interested in details, you might begin here.
Simple answer Never.
You should consider reading the official Documentation, rather than asking others to search for you. (Current Documentation leaves much to be desired, but if you read it all the answers are there.)
When I got my Pi4 (just after the release) it got hot and I installed fans. With the firmware improvements if NEVER turns on. Even a stress test takes some time to raise the temperature to 60℃.
As long as power concerns aren't an issue, consider adding a fan at any point. You can even get "silent" fans, so noise shouldn't even be an issue, except for in very quiet areas.
Even if the Pi doesn't strictly need it, it won't hurt, either. In the 30+ years I've had computers (and repaired them professionally for 15 years), I've never had a problem running one with too big a heat sink, too much airflow, or too many fans. In fact, I haven't had a hard drive fail since adding active cooling to them.
If your Pi is going through a lot of heat cycles, you could risk the solder becoming brittle or something warping to cease working. There's more reasons to having a fan than it simply overheating.