Python process Memory error

Question

I am running a Python program that is scrapping data so whenever I try to retrieve too much data (I have not been able to scrape files up to 16 MB) the programme crashes.

The error:

INFO: Got 45294 tweets for %23COVID19%20since%3A2020-03-16%20until%3A2020-03-17.
Traceback (most recent call last):
  File "/home/pi/Downloads/coronavirus2.py", line 127, in <module>
    search_term = "#NoSonVacaciones"
  File "/home/pi/Downloads/coronavirus2.py", line 77, in myfunction
    tweets= query_tweets(search_term, begindate= begin_date, enddate = end_date, poolsize=poolsize, lang= lang)
  File "/home/pi/Downloads/twitterscraper-1.3.1/twitterscraper/query.py", line 285, in query_tweets
    for new_tweets in pool.imap_unordered(partial(query_tweets_once, limit=limit_per_pool, lang=lang), queries):
  File "/usr/local/lib/python3.6/site-packages/billiard/pool.py", line 1964, in next
    raise Exception(value)
Exception: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/billiard/pool.py", line 366, in workloop
    put((READY, (job, i, result, inqW_fd)))
  File "/usr/local/lib/python3.6/site-packages/billiard/queues.py", line 366, in put
    self.send_payload(ForkingPickler.dumps(obj))
  File "/usr/local/lib/python3.6/site-packages/billiard/reduction.py", line 56, in dumps
    cls(buf, protocol).dump(obj)
billiard.pool.MaybeEncodingError:
 Error sending result: ''(<twitterscraper.tweet.Tweet object at 0x734435b0>, <twitterscraper.tweet.Tweet object at 0x726a0750>, 
 <twitterscraper.tweet.Tweet object at 0x726a03b0>, <twitterscraper.tweet.Tweet object at 0x726a0450>,
 <twitterscraper.tweet.Tweet object at 0x726a0090>, <twitterscraper.tweet.Tweet object at 0x726a0e70>,
...
 Reason: ''MemoryError()''.

Is there ay way that I can allocate more memory for this process. I do not care how many time it takes or any other processes that may be impacted.

I have a Raspbian Jessie OS updated and the memory configuration is the following:

pi@raspberrypi:~ $ cat /proc/meminfo 

MemTotal:         947748 kB
MemFree:          285512 kB
MemAvailable:     510712 kB
Buffers:           78584 kB
Cached:           193256 kB
SwapCached:         1328 kB
Active:           252492 kB
Inactive:         362552 kB
Active(anon):      99520 kB
Inactive(anon):   275632 kB
Active(file):     152972 kB
Inactive(file):    86920 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:        102396 kB
SwapFree:           1736 kB
Dirty:                20 kB
Writeback:             0 kB
AnonPages:        341888 kB
Mapped:            60048 kB
Shmem:             31944 kB
Slab:              25744 kB
SReclaimable:      11572 kB
SUnreclaim:        14172 kB
KernelStack:        1976 kB
PageTables:         4640 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      576268 kB
Committed_AS:    1269388 kB
VmallocTotal:    1114112 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
CmaTotal:           8192 kB
CmaFree:            1412 kB

code here

Thank you in advance,

Pablo

Answer 1

If you don't care about execution time, increasing your swap size from 100MB to a couple of GB will triple the amount of available memory. Projecting your current code efficiency (1 GB or RAM needed to process 16 MB of data), you should be able to process up to 48 MB of data at once.

sudo dphys-swapfile swapoff
echo 'CONF_SWAPSIZE=2048'|sudo tee /etc/dphys-swapfile
sudo dphys-swapfile swapon

Unless your data processing algorithm is really memory bound, simply writing better code could vastly reduce your memory requirements.