In general it is better to keep the case closed to not accidentally cause a short circuit. The (cpu) temperature depends on the workload. Surrounding temperature has an impact too. To check ad hoc the cpu temperature just run from command line
Divide the result by 1000 to get the value in Celsius.
Alternatively you can run:
vcgencmd measure_temp to get the temperature directly in Celsius. Details here.
I recommend to measure with case open and (some time later) with case closed to get a better "feeling" for the difference under real conditions. For constantly monitoring you could use Munin.
Whilst hitting the temperature limit is not harmful to the SoC, it will cause CPU throttling. A heatsink can help to control the core temperature and therefore performance. This is especially useful if the Pi is running inside a case. Airflow over the heatsink will make cooling more efficient. A suitable heatsink is the self-adhesive BGA (ball-grid-array) 14x14x10 mm heatsink available from RS Components.
With firmware from 12th September 2016 or later, when the core temperature is between 80'C and 85'C, a warning icon showing a red half-filled thermometer will be displayed, and the ARM cores will be throttled back. If the temperature exceeds 85'C, an icon showing a fully-filled thermometer will be displayed, and both the ARM cores and the GPU will be throttled back.
source: Raspberry Pi Documentation, Overclocking options in config.txt
Afaik heatsink and cooling fan are not part of the official starter kit, because they are not needed in most (educational) scenarios. If you simply want to prolong the lifetime of your RPi or the temperature is most of the time > 70C, I recommend a cooling fan. If this is still not sufficient, then additional heatsinks (they have relatively small impact). Both are inexpensive and easy to install.