I'd guess that you're gaining the delay due to some sort of buffering taking place or due to high CPU load. The later you can check by running the top
command.
The FluidSynth Wiki suggests three command line options you can use to reduce latency.
-r=RATE (sample rate, default 44100)
-c=NUM (number of audio buffers, default 16)
-z=SIZE (buffer size, default 64)
These go in the start_fluid
script from that page. I'd suggest setting the last two command line options to half their default values, so 8 and 32 respectively and see if that improves performance.
Edit: if sound starts to stutter, increase them until it stops and latency is acceptable. You can change them independently as well.