0

I'm trying to make my first web scraper and keep getting errors for https URLs but not http.

I am running Python 2.7.9 on RPi2 with Raspbian Jessie. I am also SSHing into the Pi on my laptop. All of this is being done at my home network.

I do not get these errors when running the code below on my laptop. Only when running it on RPi2.

I tried to find some solutions online and I found some people talking about issues with the proxy, but not sure if it applies here.

>>> import requests
>>> page = requests.get("http://www.google.com")
>>> page.status_code
200
>>> page = requests.get("https://www.timeanddate.com")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 508, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='www.timeanddate.com', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x764fbd90>: Failed to establish a new connection: [Errno 101] Network is unreachable',))
  • try urllib3 , sudo pip install urllib3 --upgrade – Dr.Rabbit Sep 30 '17 at 7:26
  • It says requirements are already up to date when I run that. – user1964692 Sep 30 '17 at 15:27
  • Are you using ipv6 by any chance? – Dr.Rabbit Sep 30 '17 at 15:52
  • I do not think so, but I followed these instructions (bneijt.nl/blog/post/enable-ipv6-on-your-raspberry-pi) to enable it. Hmmmm although before all of this, I gave my RPi2 a static IP. Does that have something to do with it? – user1964692 Sep 30 '17 at 19:50
  • I accidentally cut off some of the output error message and have added it to my OP – user1964692 Sep 30 '17 at 20:04
-1

I would try running : sudo pip install requests —upgrade

  • No luck. I'm now using latest version of requests (2.18.4) and still getting same error. – user1964692 Sep 29 '17 at 17:30
  • do you have the latest openssl packages installed aswell ? – Nicholai Cornilliac Sep 29 '17 at 22:14
  • yes i believe so. I have version 1.0.1t-1+deb8u6. I have also already ran sudo apt-get update and sudo apt-get upgrade. – user1964692 Sep 29 '17 at 23:55
1

I think that you are having timeout issues. Here is an example of 1 second timeout, which is working fine.

import requests

from requests.exceptions import ConnectionError
try:
   r = requests.get("https://www.timeanddate.com", timeout=1)
   print r.status_code
   print r.headers
   print r.encoding
   print r.text
except ConnectionError as e:
   print e
   r = "No response"
  • I tried your code and I get the following output: HTTPSConnectionPool(host='www.timeanddate.com', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConn ection object at 0x7645f910>: Failed to establish a new connection: [Errno 101] Network is unreachable',)) Using a longer timeout value doesn't help either. – user1964692 Oct 5 '17 at 22:30
  • do they even have port=443 open? – Dr.Rabbit Oct 5 '17 at 22:43
  • I just followed instructions here (techwalla.com/articles/how-to-check-if-port-443-is-open) and it looks like port 443 is open for timeanddate.com – user1964692 Oct 6 '17 at 0:45
  • remove it. Use my code then work from there – Dr.Rabbit Oct 6 '17 at 0:47
  • Remove what? I was using your code earlier. I didn't specifically use port 443 or anything. That error message is the result from using your code. – user1964692 Oct 6 '17 at 0:50
1

From the

Failed to establish a new connection: [Errno 101] Network is unreachable

error message, it seems that you are not able to connect to the server. Check that you can reach www.timeanddate.com on port 443 (HTTPS):

$ curl -I https://www.timeanddate.com

If you get something like:

HTTP/1.1 200 OK
(...)

then there may be some problem with your python installation. Else, you may have network issues. How is your PI connected to the internet?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.