30 replies to this topic

[George]

OK so I created a sweet little bot (my second overall) that loops through a text file containing keywords and scrapes the Google results then saves them to a text file.

Works GREAT! Except the results are always limited to 10. I know I can go in and do it manually at http://www.google.com/preferences but every time my compiled bot launches it's back to 10. I can of course get the bot to click the checkbox to disable suggestions which allows you to move the slider to 100. But for the life of me I can't figure out how to get ubot to move the slider.


My ultimate goal is to have a bot I can set to run daily and have it go in and set it to 100 results then do its thing. Any suggestions.

EDIT: Thanks Kreatus. I guess I have to o something to add the fact that I have a license to my profile here but it's not exactly readily available information around here or in the welcome email...

In any case I can't download what you attached (I can't even reply to my own thread.) but I did figure a crude method out. I simply put a click command in a loop and selected the element "<style="width:6px">". Then I just clicked advanced, made sure it was set to Left Mouse Click and changed Click Using Location to YES. Ran the loop 5 times. It worked like a charm.

Edited by George, 19 February 2012 - 08:41 PM.

[Kreatus (Ubot Ninja)]

Hi, check this code below. This is the google scraper source I uploaded on bot bank but hasn't approve yet.
It will scrape up to 1000 google result urls.

 google scraper.ubot   1.03K   25 downloads

[Duane]

Hey Kreatus,

Thanks for the bot but it doesn't work for me, it freezes up when it gets to:

loop(5) {
click(<image="___IMAGE___0___IMAGE___">, "Left Click", "No")
}

Not sure if anyone else is having the same problem but I wanted to let you know.

[a2mateit]

Freezing on loop(5) {
click(<image="___IMAGE___0___IMAGE___">, "Left Click", "No")
} for me to

[Kreatus (Ubot Ninja)]

Not sure what happened to my code but it should be an image on that click command..

Here's the updated one  google scraper.ubot   2.18K   19 downloads

[JohnB]

George,

send me your username and I will be able to lift the forum restrictions for you.

John

[Duane]

Ah... much better!

Thanks!

[George]

George,

send me your username and I will be able to lift the forum restrictions for you.

John

The person that called me to validate the license took care of me.

Thanks.

[turbolapp]

I'm trying to get 100 results as well but the methods talked about here aren't working for me. Anyone have a current work around?

Edit: nm. When trying George's method I missed click using location "yes". Wow that is some impressive duct tape!!

[Kreatus (Ubot Ninja)]

I'm trying to get 100 results as well but the methods talked about here aren't working for me. Anyone have a current work around?

Edit: nm. When trying George's method I missed click using location "yes". Wow that is some impressive duct tape!!

Check this  google scraper.ubot   2.64K   6 downloads this should fix the problem from my last upload.

[turbolapp]

Check this  google scraper.ubot   2.64K   6 downloads this should fix the problem from my last upload.

Your bot doesn't work. Gets shut down actually.

But's it's fine I'm doing George's work around.

HOWEVER

I can't for the life of me click the goddamn save button. Any ideas?

[Kreatus (Ubot Ninja)]

Your bot doesn't work. Gets shut down actually.

But's it's fine I'm doing George's work around.

HOWEVER

I can't for the life of me click the goddamn save button. Any ideas?

Ok. Well it is working perfectly on my side.

[turbolapp]

I didn't see where it clicked the save button. How are you doing this?

[Kreatus (Ubot Ninja)]

By clicking the side of the save button  3-23-2012 12-52-16 AM.png   4.7K   7 downloads since every country have different save button text.

[turbolapp]

I'm having the same problem that the two posters above you had. When I copy your module and paste it into mine I get what they get: a frozen bot.

When I just run your bot as is it freezes.

When I try to examine the element to try to find this image module of yours, all i get to work with is: <class="goog-inline-block jfk-button jfk-button-action jfk-button-hover"> which does nothing.

At my wits end here over one stupid button.

[Kreatus (Ubot Ninja)]

I'm having the same problem that the two posters above you had. When I copy your module and paste it into mine I get what they get: a frozen bot.

When I just run your bot as is it freezes.

When I try to examine the element to try to find this image module of yours, all i get to work with is: <class="goog-inline-block jfk-button jfk-button-action jfk-button-hover"> which does nothing.

At my wits end here over one stupid button.

Not sure why it's freezing at your end but it is running normally on my computer.
Try this  googlescraper.ubot   4.24K   8 downloads

[turbolapp]

Alright, looks like I finally found a way to click it:

click(<outerhtml="<div role=\"button\" class=\"goog-inline-block jfk-button jfk-button-action\" tabindex=\"0\" style=\"-webkit-user-select: none; \">Save</div>">, "Left Click", "No")


Wasted a morning on it, but I win cause I figured it out. Right?...Right?

Edit: Kreatus...maybe it's something to do with your dev licences vs my standard? dunno. but thanks for trying.

[turbolapp]

Ahhh the google gods tease.

As you stated above it won't work in other countries.

So I guess I'll spend the rest of my day on this because I'm just too bullheaded to freakin stop.

(btw, your latest bot still has the same problem. I get that pop up from ubot telling me ubot has stopped working and must be closed. I think it probably is the different versions problem because that image module is foreign to me)


Anyways, I'm attaching the bot I have so far if anyone wants to take a gander at it.

Attached Files

•  help.ubot   635bytes   5 downloads

[a2mateit]

Turbolapp,

I just took a look at your code and this is working for me to click save:

click(<innertext="Save">, "Left Click", "No")

And here is your bot with it set up:
 change google preferences to 100.ubot   808bytes   2 downloads

HTH
Justin

[turbolapp]

Still the fundamental problem of not being able to scrape in other countries. Which is why Kreatus went with the image and why I would if I could.

Edit: I think maybe this time I got it by making my previous solution a wild card just taking out save. (I thought I had already done this but obviously not.) anyways tried it on a russian proxy and it worked just like on the us proxy. So fingers crossed.

Thanks for watching me ramble all day