I know there are some tools for MATLAB-RPi connection e.g. using simulink add-ons and blocks. But is it possible to install MATLAB directly on RPi?
No, it is not possible. MATLAB is only available for the Intel x86 architecture while the Raspberry Pi uses an ARM processor.
However you can install Mathematica which has much of the functionality of MATLAB, including alternatives to many of the MATLAB toolboxes, and is available for the RPi for free.
You can also install GNU Octave using
apt-get install octave. Octave is a MATLAB clone.
The J language (an APL variant) is also available for the Raspberry Pi and is a viable alternative for the base MATLAB language (though not the toolboxes). R and Python + numpy are great for number crunching too.
You can use the Raspberry Pi as a target board in simulink and Matlab. Have a look at: