You can pull the FAT partition out of your Linux system without a problem but you would need to do some stuff with a loader for Linux.
Okay the basic here is the ARM CPU(s) are co-processor to the GPU and the GPU needs a FAT partition to load the ARM code up before releasing them.
The GPU has its own FAT reader and it reads four mandatory files which must reside on a FAT partition
kernel.img is our user code and you can basically do whatever you want from there.
So the bottom line here is there has to be a FAT on the card but no you don't have to be able to read FAT in your code. If you look at any bare metal code on the Pi I assure you that almost none have a FAT partition reader. The thing that reads the FAT for them is the GPU.
So long as Linux is happy to leave the FAT partition alone and you have a loader that can pass into linux, then Linux itself requires no understanding of the FAT system.
I suspect you will have to put the driver tree files on your Linux partition and load them at the startup of linux because I suspect Linux will need them.
If I was going to do it I would compile code for the linux loader for the Linux file system. Place it on the fat partition and write a small piece of bare metal code that simply places the file at
0x8000 onwards in the Pi and then jump the
0x8000 and it should run straight into linux without ever knowing about FAT. Any reasonable bare metal coder would be able to write that code and know what it does.