Please read manuals. There is no 'AT' + '\r\n', use "AT\r" instead. No need for New Line '\n', only Carriage Return '\r'. Also, before any other commands, SIM800 is by default working at 115200 baudrate, so "AT\r" + "AT+IPR=9600\r" or any other baudrate is a MUST been sent to it.
This is a baud rate problem. You're getting a lot of x and f characters which have ASCII codes of 01111000 and 01100110 - note all bits inside those are repeated twice. The effective baud rate of your Pi should be twice as low as what you think: if your terminal is set to 115200, try setting it to 57600.
Perhaps reading the messages you're receiving (using ...
If hardware loopback works, then it could be clock issue.
When using the analog tv out via the 3.5 mm jack on the RasPi4, then the clock gets reduced (see the RasPi documentation about overclocking). The UART seems to work properly only with HDMI output.
There's nothing wrong with your approach for individual text files. For a binary file (or a large number of files/directories which you'd pack together with tar/gzip), you can use base64 to encode them as text:
Receiver (Pi over picocom):
cat | base64 -di > file
Sender (another terminal on the PC):
cat file | base64 > /proc/`pidof picocom`/fd/0
Although old fashioned, Picocom supports filetransfer, x-modem, y-modem, z-modem and ascii-xfr. If I where you, i'd go for z-modem. The package you are looking for is lrzsz (and notzmodem as I stated earlier). Note that it operates a bit different from scp.
wiringSerial is part of wiringPi.
wiringPi is an "Arduino" like C library to control the GPIO.
wiringSerial is a simple wrapper around the underlying Linux serial driver. It simplifies some operations. Look at the source code if you need details.
Note that wiringPi is deprecated (i.e. no longer being developed or supported by its author).