How can I scroll down a webpage using Selenium with VBA?

  • Hi Ranjith!! Hope you are doing fine. I got a problem accomplishing my little piece of code which is slightly different from what I usually ask for. I've used javascript function within vba code which is written in combination with selenium. I tried to scroll to the downmost portion of a lazy-loading webpage using the below code. However, upon running my script I can get all the links available in that page. All I need to know is whether the way I used "for x" loop is the ideal one or I should follow something else. If what I'm doing is right then one thing i need to know for the clarity which is: why the loop has to be "0 to 200" to get the 1000 links from that page and how I can make it without hardcoding any number as in, 200 in this case to create a loop until i reach the dead end. Here is the code I've written so far:

    Sub Going_down()
    Dim driver As New WebDriver
    Dim posts As Object, post As Object
    driver.Start "chrome", ""
    driver.get "/list/"
    driver.wait 500
    for x = 0 to 200
         driver.ExecuteScript("window.scrollTo(0, document.body.scrollHeight);")
         driver.wait 500
    next x
    Set posts = driver.FindElementsByXPath("//li[contains(concat(' ', @class, ' '), ' small-12 ')]")
    For Each post In posts
        i = i + 1
        Cells(i, 1) = post.FindElementByXPath(".//a").Attribute("href")
    Next post
    End Sub

  • administrators

    I would just call their API instead of using selenium. BTW Why are you scraping all these data?

  • Thanks for your response. The data of that site more specifically any site is not important to me at all. However, what does important to me is that whatever the structure of a site is, I can deal with that. That is why I rove different sites and try to understand their pattern. That's it.