UTF-8 support on Raspberry Pi

Question

I want to run a python program that receives the data through serial communication and saves it to a file, but after receiving the data this error shows up. Any idea? (code runs ok in Windows but in Raspbian OS I have this error)(the data is in JSON format)

error:

Traceback (most recent call last): File "/home/pi/Desktop/ReadFromPort.py", line 11, in line = ser.readline().decode("utf-8") UnicodeDecoderError: 'utf-8' codec can't decode byte 0xbd in position 2: invalid start byte

Python code:

from serial import *
from serial.tools import list_ports
import os

ser = Serial('/dev/ttyS0',115200, timeout = 1)


with open("output1.txt","w",encoding="utf-8") as file:
    while True:
        line = ser.readline().decode("utf-8")
        if(line != ""):
            #print(line)
            file.write(line)
            file.flush()
            os.fsync(file)

Answer 1

File is not in utf8 format, so the decoder is seeing an invalid (i.e., non-utf8) byte sequence. 0xbd is the ½ character in Latin-1, utf8 represents it as the two-byte sequence \xc2\xbd

Answer 2

Type of ser.readline() is bytes.

bytes.decode(encoding='utf-8') return a str if and only if there is not a UnicodeError risen. It is this in fact that appens in your line

line=ser.readline().decode('utf-8')

So, you must give a good value to the encoding parameter.

You'll find the different encoding here. If it is a latin-1 (for instance), you must have the following code :

with open("output1.txt","w",encoding="utf-8") as file:
while True:
    line = ser.readline().decode("latin-1")
    if(line != ""):
        #print(line)
        file.write(line)
        file.flush()
        os.fsync(file)

To improve code quality, you can take avantage of catching exceptions ;)