How to transfer large amount of data (100Kbytes) over SPI?

Question

Related/previous question here. I'm looking to do a data logging project that involves transmitting data from an FPGA to a Raspberry Pi 4 through SPI. The FPGA will transfer 250,000 bytes approximately every 7 seconds. I would like for the Pi to read this data and convert/organise it as needed. Ideally the data would be transferred within a second in order to give the Pi enough time to process the data, so I'm looking to have a SPI speed of at least 2Mbit/s. So far, I'm using an STM32 to provide dummy values (up to 100Kbytes) and to just get the SPI working on the Pi. Because the Pi won't know when data is ready (and it HAS to be the master on the SPI bus) the slave device will generate a separate signal on a GPIO pin just to let the Pi know that it needs access to the SPI bus. I'm using python spidev. The code:

#a simple test for spi

import spidev
import time
import RPi.GPIO as GPIO

GPIO.setmode(GPIO.BOARD)
GPIO.setup(7, GPIO.IN)
#using spi0
bus = 0

#device is chip select.
device = 0

#enable
spi = spidev.SpiDev()

#open a connection to a specific bus and device (CS pin)
spi.open(bus, device)

#set spi speed and mode
spi.max_speed_hz = 5000000
spi.mode = 0

#msg = ["Hello! "]
msg = [72, 101, 108, 108, 111, 33, 32]
msgRx = 0;
print ("python spi test program with STM32")
#GPIO.output(7,0)
time.sleep (1)
while True :
    
    if GPIO.input(7) == 1:
        time.sleep(0.2)#needed to ensure that there aren'tmultiple triggers within the 100ms pulse
        #msgRx = spi.readbytes(100000)
        msgRx = spi.xfer3(100000)
        print ("Message received")
        print (msgRx)

So the code sets up the GPIO and SPI accordingly and then it sits in a loop, polling a pin waiting for the slave device (STM32F4 in this case) to be ready. The pin is set high every 3 seconds for a period of 100ms. As you can see in the code, I want to read 100 Kbytes. This code works fine if I want to just read a few bytes, but it doesn't work with larger data transfers. It seems to just read in the first few hundred bytes. So, to clarify, the bus speed seems to be OK but the amount of data I can transfer in one go is the issue.

Now my question: How can I get the raspberry pi to accept a larger data transfer over SPI (>100Kbytes)? Do I need to split the data up in to packets or smaller chunks? If so, how can I do that in spidev? Is there a maximum throughput for SPI in any one transaction? Any help would be appreciated.

Update: For anyone who may come across this question in the future, follow @joan 's advice for extending the buffer size. If you are specifically using spidev however, see here for the best documentation I've seen for spidev. Be warned it is poor. Page 10 states that spidev ignores the buffer you set in boot/cmdline.txt unless you specifically use xfer3. You are limited to a buffer of 4096 if you only want to read. You must use the read/write mode.

Answer 1

I think the default Linux SPI driver buffer size is 4096 bytes.

You will need to search for methods of making the buffer larger.

On the Pi set the buffer size by editing /boot/cmdline.txt and adding the following at the end of the (single) line.

spidev.bufsiz=250000

where 250000 is your preferred buffer size.

The following lg script will then read 65536 bytes per transfer (65536 is the maximum size supported by lg).

#!/usr/bin/env python

TRIGGER=5

TRANSFER=65536

import sys
import time
import lgpio as sbc

def cbf(chip, gpio, level, tick):
   if level == 1:
      count, data = sbc.spi_read(spi, TRANSFER)
      print("read {} bytes".format(count))

handle = sbc.gpiochip_open(0)

if handle < 0:

   print("Can't open /dev/gpiochip0")
   exit()

# got a handle, now open the GPIO for alerts

err = sbc.gpio_claim_alert(handle, TRIGGER, sbc.BOTH_EDGES)

if err < 0:

   print("GPIO in use {} ({})".format(TRIGGER, sbc.error_text(err)))
   exit()

cb_id = sbc.callback(handle, TRIGGER, sbc.BOTH_EDGES, cbf)

spi = sbc.spi_open(0, 0, 4000000)

while True:
   time.sleep(1)