According to the Pi 4 Bootloader Configuration documentation you can patch the boot ROM to preset various configuration values, and one of them is the TFTP server address using
TFTP_IP (since EEPROM version 2020-04-16).
You can also configure a static IP address to avoid relying on DHCP just to reach the TFTP server.
TFTP_IP and the following options are set then DHCP is skipped and the static IP configuration is applied.
The static IP options are
You'll need to follow the instructions at that link as they are a bit long to repost here, but the gist of it is:
cp /lib/firmware/raspberrypi/bootloader/beta/pieeprom-2019-09-23.bin pieeprom.bin
rpi-eeprom-config pieeprom.bin > bootconf.txt
# Edit bootconf.txt and set TFTP_IP=126.96.36.199
rpi-eeprom-config --out pieeprom-netboot.bin --config bootconf.txt pieeprom.bin
sudo rpi-eeprom-update -d -f ./pieeprom-netboot.bin # Flash updated boot ROM
Basically you extract the boot ROM config into a text file, edit it, then merge it back to the ROM file and reflash it.