0

I am about to drive myself mad with this problem. After a few days, I have gotten my Raspberry Pi up and running with Selenium using Chromium and its webdriver. It pulls up the webpage and even scrolls has I have instructed it to do. The only thing it is not doing is finding the class I'm specifying. It's a little complex because I am trying to scrape the Twitter Reply section which has endless scrolling, therefore I have to use Selenium to get the page_source for each scroll for it to grab the usernames. It works on my Desktop when I run the script, but not on my pi. What kinds of things could make this happen? Here is some of the script:

import requests, time, botometer
from bs4 import BeautifulSoup
from selenium import webdriver
from lxml import html
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys

url = ("https://twitter.com/realDonaldTrump/status/1265827032707072000")
driver = webdriver.Chrome(
    'chrome/driver/here')

def scroll_to_bottom(driver):
        # Empty list for all the usernames scraped
        FULL_LIST = []
        bots = []

        old_position = 0
        new_position = None

        while new_position != old_position:
            # Get old scroll position
                old_position = driver.execute_script(
                ("return (window.pageYOffset !== undefined) ?"
                " window.pageYOffset : (document.documentElement ||"
                " document.body.parentNode || document.body);"))

                # Scrapes usernames from current scrolls
                html = driver.page_source
                soup = BeautifulSoup(html, features='lxml')
                usernameSoup = soup.find_all(
                "div", class_="css-901oao css-bfa6kz r-111h2gw r-18u37iz r-1qd0xha r-a023e6 r-16dba41 r-ad9z0x r-bcqeeo r-qvutc0")
                list_of_usernames = [x.text for x in usernameSoup]

                # Prints current loot of usernames
                text = ', '.join(list_of_usernames)
                print("SCROLLINGS LOOT:", text)
                print("LEN:", len(list_of_usernames))

                # Extends master list of usernames
                FULL_LIST.extend(list_of_usernames)

                # Deletes any duplicates in FULL_LIST
                FULL_LIST = list(dict.fromkeys(FULL_LIST))

                # Sleep and Scroll
                time.sleep(5)
                driver.execute_script((
                "var scrollingElement = (document.scrollingElement ||"
                " document.body);scrollingElement.scrollTop ="
                " scrollingElement.scrollHeight;"))
                # Get new position
                new_position = driver.execute_script(
                ("return (window.pageYOffset !== undefined) ?"
                " window.pageYOffset : (document.documentElement ||"
                " document.body.parentNode || document.body);"))
                print("FULL LIST: ", len(FULL_LIST))

driver.get(url)
scroll_to_bottom(driver)
driver.quit()

  • 1
    I have the impression that the CSS classes are not constant. I've tried your code and there are variations. I see three possibilities: using a looser CSS selector, using an XPATH selector so you don't depend on class names, or use the Twitter API. The API would be a much better option provided it can return all the items you need. Then you can use the requests module instead of Selenium. By the way you are importing the requests module but as far as I can tell it's unneeded. – Anonymous Jun 2 at 23:37
  • Thank you! I have tried the Twitter API but as far as I can tell, it does not return the list of users that are replying to tweets. It seems like the Twitter API only returns basic information specifics about a specific account. – snoofle Jun 3 at 0:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.