4 replies to this topic

[Jon12345]

When I go to Twitter, I see 20 odd tweets ready to scrape. But if I scroll to the bottom of the page, it keeps adding more Tweets. Before, I am sure there was a button that you used to click for more tweets, right?

So what can you do now? If I want to scrape I sizeable page of tweets, I figure I will need to load the page and then scroll that page a few times, then do the scrape.

How do I get ubot to scroll so more tweets are added?

Thanks,

Jon

[turbolapp]

run javascript
location.reload()

Assuming you're searching through popular results, like trends or something, you need only to refresh the page for new results.

[JohnB]

I just logged in and it looks like it just keeps loading and loading...you may just have to put in a significant delay and wait for them (i.e. 60 seconds or more)

John

[LoWrIdErTJ - BotGuru]

run javascript
window.scrollTo(0,document.body.scrollHeight);
delay or wait finish
run it again
window.scrollTo(0,document.body.scrollHeight);
delay or wait finish

repeat process as many times as you want till enough are showing.

Have fun.

TJ

[amul]

run javascript
window.scrollTo(0,document.body.scrollHeight);
delay or wait finish
run it again
window.scrollTo(0,document.body.scrollHeight);
delay or wait finish

repeat process as many times as you want till enough are showing.

Have fun.

TJ

I was trying to figure out how to do get javascript to load this..
Now I've to do some regex for friend ids.. to message.

thanks TJ!!