3

I'm thinking of getting a raspberry pi to do some scraping work so i can go out while everything continues to run while at home. Is there any reason why Scrapy would be a bad idea for a Raspberry Pi setup? Potential issues i was thinking would be WiFi speeds, performance, and system requirements.

  • Have you had any success with this as we are looking at doing this on a raspberry pi to scrap our competitors sites for prices etc. Does it work okay and how easy was it to install if you got it working? – user8031 Jun 18 '13 at 14:27
3

There is no particular limitation on the pi with regard to network connection speeds, wifi or otherwise. You'll get whatever your adapter provides.

I'm sure the pi will be able to keep up parsing html as fast as the internet can realistically deliver it. Maybe it will start to fall behind if you are doing it with local network data that can be accessed consistently at very high speed, or the scraping is very convoluted, but in general the pi is probably ideal for this.

From the looks of the "scrapy" page it is pure python, portable to linux, and shouldn't pose a problem requirements wise.

| improve this answer | |
  • I better get my order placed then! All the GPIO stuff looks fun too. Thank you. Much appreciated! – john Mar 26 '13 at 17:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.