I have a RPi4 now, have cooling sinks mounted on the processors and it runs pretty smooth, but sometimes I worry about the temperature development of my processor. At what level should I worry, and consider a fan?
You needn't worry at all if your only concern is temperature. All RPi models have a built-in closed-loop thermal control system that will prevent your RPi from overheating. This system operates by reducing the operating frequency and core voltage when the temperature reaches set limits. These frequency & voltage reductions are called throttling. Throttling reduces the power consumed in the SoC, which reduces heat/temperature - it also reduces performance of the system.
Understanding the operation of the RPi thermal management system might lead one to reconsider the question you have posed. You can see now that since the system manages temperature, performance is the variable one must reckon with. And so I say the answer to your question is this:
Consider a fan only when your RPi is being throttled to the point that its performance becomes insufficient to meet your needs.
Wrt the relationships between temperature and performance, there are numerous other details you may wish to explore. While the RPi's thermal management system is implemented in proprietary firmware, there are user-accessible configuration parameters available in the RPi that allow you to change some aspects of its performance. If you're interested in details, you might begin here.
Simple answer Never.
You should consider reading the official Documentation, rather than asking others to search for you. (Current Documentation leaves much to be desired, but if you read it all the answers are there.)
When I got my Pi4 (just after the release) it got hot and I installed fans. With the firmware improvements if NEVER turns on. Even a stress test takes some time to raise the temperature to 60℃.
As long as power concerns aren't an issue, consider adding a fan at any point. You can even get "silent" fans, so noise shouldn't even be an issue, except for in very quiet areas.
Even if the Pi doesn't strictly need it, it won't hurt, either. In the 30+ years I've had computers (and repaired them professionally for 15 years), I've never had a problem running one with too big a heat sink, too much airflow, or too many fans. In fact, I haven't had a hard drive fail since adding active cooling to them.
If your Pi is going through a lot of heat cycles, you could risk the solder becoming brittle or something warping to cease working. There's more reasons to having a fan than it simply overheating.
Frequency Management and Thermal Control
All Raspberry Pi models perform a degree of thermal management to avoid overheating under heavy load. The SoCs have an internal temperature sensor, which software on the GPU polls to ensure that temperatures do not exceed a predefined limit; this is 85°C on all models. It is possible to set this to a lower value, but not to a higher one. As the device approaches the limit, various frequencies and sometimes voltages used on the chip (ARM, GPU) are reduced. This reduces the amount of heat generated, keeping the temperature under control.
When the core temperature is between 80°C and 85°C, a warning icon showing a red half-filled thermometer will be displayed, and the ARM cores will be progressively throttled back. If the temperature reaches 85°C, an icon showing a fully filled thermometer will be displayed, and both the ARM cores and the GPU will be throttled back. See the page on warning icons for images of the icons.
For Raspberry Pi 3 Model B+, the PCB technology has been changed to provide better heat dissipation and increased thermal mass. In addition, a soft temperature limit has been introduced, with the goal of maximising the time for which a device can "sprint" before reaching the hard limit at 85°C. When the soft limit is reached, the clock speed is reduced from 1.4GHz to 1.2GHz, and the operating voltage is reduced slightly. This reduces the rate of temperature increase: we trade a short period at 1.4GHz for a longer period at 1.2GHz. By default, the soft limit is 60°C, and this can be changed via the temp_soft_limit setting in config.txt.
The Raspberry Pi 4 Model B, continues with the same PCB technology as the Raspberry Pi 3 Model B+, to help dissipate excess heat. There is currently no soft limit defined.
source: Raspberry Pi documentation