I'm Korean and had made Korean-web crawling program using Python3.

I checked that it runs on OSX.

And I tried to port this program to my Raspberry pi 2 (Linux raspberrypi 4.1.16-v7+)
(Actually, I only change the code to use Selenium webdriver Firfox() in Raspbain. In OSX, I use PhantomJS)

When I start run, it occurs errors like this :

Traceback (most recent call last):
  File "my_snu.py", line 106, in <module>
    message = my_snu.all()
  File "my_snu.py", line 94, in all
    contents += self.notice.__name__ + '(http://community.snu.ac.kr/bbs/bbs.message.list.screen?bbs_id=460)' + '\n' + self.notice('no')
  File "my_snu.py", line 76, in notice
    return self._get_contents('http://community.snu.ac.kr/bbs/bbs.message.list.screen?bbs_id=460', default_next_page)
  File "my_snu.py", line 66, in _get_contents
    table_component['title'].encode("latin1", 'ignore').decode("euc_kr")) + '\n')
UnicodeDecodeError: 'euc_kr' codec can't decode byte 0xb7 in position 14: illegal multibyte sequence

This is the part of the code which occurs error :

content += ('{}  {}'.format(table_component['when'],
                            table_component['title'].encode("latin1", 'ignore').decode("euc_kr", 'ignore')) + '\n')

I have no idea why this occurs because both OS(MAC OSX, Raspbian) uses Python3 and sys.getdefaultencoding() returns 'utf-8'.

p.s I already ask question about this problem last time in stackoverflow : https://stackoverflow.com/questions/35114240/python3-encode-decode-issue?noredirect=1#comment57992141_35114240

But need more help from Raspberry pi users. Thanks :)

  • I would need more code to be able to help sadly but could there be differences in how any imports are built? – Davies-Barnard Feb 3 '16 at 7:18
  • how did you run this on the Pi? – Steve Robillard Feb 3 '16 at 7:19
  • @Davies-Barnard I didn't upload the code because the site that I'm trying to crawl is only allowed for authorized user. Oh I forgot to say that this site language is Korean. (I'm also Korean :) ). I made a huge mistakes. – user3595632 Feb 3 '16 at 8:05
  • @SteveRobillard ?? just python program.py.... in my terminal. – user3595632 Feb 3 '16 at 8:06
  • Try python3 program.py you ran it with python 2.7.x not python3 hence the unicode errors. – Steve Robillard Feb 3 '16 at 8:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.