1

I am new to posting questions on this site so forgive me if this isn't in the best place or if I do something wrong.

I have two Pi's and I am trying to capture images in a loop (giving me a video), with the Raspberry Pi camera and send each through a socket to the other Pi and display them with pygame. I will also need low latency.

I have tried many different methods and think I have gotten fairly close. The code I have right now is posted at the bottom. There are however, a lot of other things I have tried. For example, I tried using the PiCamera module to capture an image and then make it a PIL image right away (instead of capturing an image with pygame). The code below seems to be fairly close but I can only seem to get the delay to about 5 seconds.

I also tried using UDP but could not get time image.tostring small enough to send through the socket.

I realize there are other ways to do this, like mjpg-streamer but I wanted something a bit less complicated that I could understand.

Any help would be appreciated!

server:

import socket
import pygame as game
import sys
from PIL import Image

port=9012
address="192.168.1.10" #server's ip
size=width, height= 640, 480
scale=width, height=40, 10
timer = 0
previousImage = ""
image = ""
message=[]

#establish socket connection
s=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((address, port))
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)



#initialize pygame
game.init()
screen=game.display.set_mode(size)
game.display.set_caption('Socket Raspi Camera Viewer')
#clock = game.time.Clock()

#listen on the port
s.listen(1)


#main loop
while 1:
  for event in game.event.get():
    if event.type == game.QUIT:
      sys.exit()

  #accept socket connection
  connection, addr= s.accept()

  #Recieve data from the server:
  data=connection.recv(1024000)
  print(sys.getsizeof(data))

  #store the previous recieved image incase the client fails to recive all of the data for the new image
  previousImage=image

  #We use a try clause to the program will not abort if there is an error:
  try:

    #We turn the data we revieved into a 120x90 PIL image:
    #image = Image.fromstring("RGB",(120,90),data)

    #We resize the image to 640x480:
    pil_image = Image.fromstring("RGBA", size, data)
    pil_image= pil_image.resize(size)
    new_data=pil_image.tostring()
    image=game.image.fromstring(new_data, size, "RGBA")

  except:
    #If we failed to recieve a new image we display the last image we revieved:
    print("falied to recieve image")
    image = previousImage

  #Set the var output to our image:    
  #output = image

  #We use PyGame to blit the output image to the screen:
  screen.blit(image,(0,0))

  #We update the screen:
  game.display.flip()`

client:

import pygame as game
import pygame.camera
import sys
import time
from PIL import Image


port=9012 
address="192.168.1.10" #server's ip
size=width, height=  640, 480
scale=width, height= 40, 10
timer=0

game.init()
game.camera.init()
screen=game.display.set_mode(size)

camera=game.camera.Camera("/dev/video0", size )
camera.start()
time.sleep(1)
while True:
  if timer < 1:
    #connect to the server
    s=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.connect((address, port))

    #get image
    image=camera.get_image()

    #convert to pill image to scale
    string_image=game.image.tostring(image, "RGBA",False)
    pil_image=Image.fromstring("RGBA",size,string_image)
    pil_image =pil_image.resize(scale)
    buffer=pil_image.tostring()

    #print size of buffer
    print(sys.getsizeof(buffer))
    s.sendall(buffer)
    #time.sleep(0.01)
    s.close()
    timer=30
#

Stream without resizing: client-

while True:
  image=camera.getimage()
  buffer=pygame.image.tostring(image)
  socket.send(buffer) 

Server:

while True:
   message=[]
      while True:
         string=socket.recieve(1024 * 1024)

         if string == '':
            break

         else:
            message.append(string)

      full_string="".join(message)
      image=game.fromstring(full_string)
      game.blit(image, (0,0))
      game.display.update()
0

I must confess that I know nothing about image processing from the Pi camera and I've only played around a little bit with Python. Yet some things look suspicious about your code.

You seem to be creating a socket connection, sending one image (still frame) down the pipe, closing the socket, then repeat. You don't want to be doing it this way. Too much overhead and I think you will quickly run out of socket numbers. On the server side, the part with the listen and accept: you only want to do this ONCE. i.e. Do your listen and wait for the client side to connect, then server accepts, THEN you want to stay in your loop sending and receiving image frames. Only when you are all done do you want to close the socket. The way you are doing it (I think) is creating tremendous overhead in the comms portion of your system. You may want to invent a little protocol for yourself so that after processing a received image frame, the server sends (for example) "OK" back to the client to say "last frame good, ready for next". The client waits for "OK" before sending another frame.

Likewise, with the image processing, again please forgive me if I'm talking out of turn because I don't know anything about this specifically, but I think you want to get the data from the camera and transmit it immediately, not be messing around resizing, converting to/from strings, etc, on the client side. Let the server do the needed transformations because he is the guy who is going to display it. In my opinion, as soon as you camera.get_image you should have the data for one image sitting in your buffer "image". SEND IT. As long as you know the length of your data, sockets should have no problem sending it as binary data. You don't need to convert to string (unless that is the only way you can find out the length). All those conversions and resizes are expensive and that has to be killing your performance. Just get the data from the camera and do as little as possible and then send it. Let the server side monkey around with it.

Here again, DON't CLOSE YOUR SOCKET. Keep using it!

That's my two cents. Good luck.

  • I have considered that but couldn't seem to get it to work that way. It does make sense. This version of my code has the resizing other versions did not need it. The client resizes it so it can be sent and received through the socket with out having to add a whole new while loop, in the server, for receiving the huge image string. Are you saying send the image without converting it to a string? I was under the impression that was impossible. I will however keep working on sending the data with one socket connection. Thanks for the help. – user67468 May 21 '17 at 23:04
  • The "image" should just be a series of bytes. As long as you know the length, you should be able to send it through a socket. The only reason I could see that would mandate converting it to a "String" would be if your communications channel could not support binary data (socket should be able to handle it) or, for example, storing in a database which could not handle binary data. Now I understand that you were trying to reduce the volume of data with your resize. That may be a reasonable thing to try. Any idea how big (bytes) is a single image before and after a resize? – MartinMarty May 23 '17 at 3:47
  • sending an image like this: image=camera.capture() and then socket.send(image) did not work. It says "must be string or buffer, not pygame surface". Using sys.getsizeof(buffer) the full size image is about 921621 and the resized image is 1621 (these are the string versions of the image). Doing sys.getsizeof(image) gives about 28, but after resizing i am getting 36. I must be doing something wrong. Also, sys.getsizeof() may be the wrong function, i don't know what else to use. The main problem i am having with resizing is when blit ing the image on the server side it looks weird. – user67468 May 23 '17 at 17:14
  • Ok, given what you said, I'd have to agree that converting to string is necessary to get a plain ol' "series of bytes" that I'm talking about. Can you imagine how much of your image you must be losing going from 920K to 1621 bytes? Something seems wrong there... But, even at 920K, on 100Mb LAN, seems like we could probably send the buffer in (let's say) .5 seconds max, allowing for TCP/IP stack to bust it down into packets, add a lot of protocol and re-assemble on the other end. I'd concentrate on sending the UN-resized buffer first, make sure you rcv same # bytes on server side, then refine. – MartinMarty May 24 '17 at 7:48
  • Thank you so much for your help, i really appreciate it. I edited the original question to contain a very simple streaming method that pretty much works (not every detail is there but you should get the idea). As you can see it takes an inner loop to receive the image, then the outside loop continues and blits the image to the screen. The version above does not have a socket connection for every image but WITH them i am able to get the latency to about 5 seconds. I have somewhat got it working without them but the latency more than doubles to about 12 seconds. Both codes are fairly similar. – user67468 May 24 '17 at 16:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.