I have the same issue on Debian Stretch (not Raspberry). Unfortunatly I dont have a solution yet. It could be a bug in the Debian repository.
Debian Stretch comes with:
OpenSSL 1.1.0e 16 Feb 2017
However, Debian seems to supply a library for 1.0.2:
# dpkg -S pkcs11.so
There also seems to be a bugreport for this issue, but it looks like it will not be fixed any time soon. See Debian Bug https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=846548
To resolve this issue I compiled the PKCS#11 engine myself. I downloaded it from https://github.com/OpenSC/libp11/
Please note that besides the usual compilation tools (apt-get install build-essential autoconf automake) you also need package pkg-config. Otherwise the installation fails.
Using the newly compiled driver I managed to get it working:
# OPENSSL_CONF=./hsm.conf openssl engine
(dynamic) Dynamic engine loading support
(pkcs11) pkcs11 engine