Jump to content
UBot Underground

My scraper bot is freezing up.....


Recommended Posts

I have a simple bot I made that scrapes for two pieces of data from every url I add to a list for ubot to navigate to.

 

I'm actually having two issues with this little bot. The first (and my initial reason for posting this) is that every 20th or so url ubot navigates to, it gets stuck on that url and won't continue on to the other thousand or so pages I want it to scrape.

 

Second, it's basically a workaround of what I really want to acclomplish with this tool. One of the pieces of data I'm trying to scrape shows up maybe 50 to 75% of the time in the meta decription of these pages, so I've been settling for scraping that instead of what I really want to scrape on the page. The second thing I'm scraping is a specific url off each page, but I can't figure out how to get the specific one I want so I'm scraping them all which is usually about 10 extras per page and then I (sort of) filter them out.

 

I'm uploading a screenshot of what I have so far with my ubot code and an example of the two pieces of data I'm scraping, so if anyone has any suggestions on any of these issues that would be great. Thanks!

 

First half of code: post-1210-12858937072068_thumb.png

 

Second half: post-1210-12858937360648_thumb.png

 

Example of pages I'm scrape and what I'm scraping: post-1210-12858938697819_thumb.png

Link to post
Share on other sites

Ok, so I tried to run it the way meter suggested, scraping just the meta description. I ran the bot and tracked it's results, and it got stuck on the 55th url.

 

I think it's made it that far before. I'm still looking for a solution for this issue, so if anybody has any other suggestions about how to solve this problem that would be a big help.

 

I know this software is capable of doing tasks like this without freezing up, because I made a different simple bot that's been running for the last 2 weeks or so non stop without freezing!

Link to post
Share on other sites
  • 3 years later...

Hey  Josh, 

 

Wanted to let you know, that I encountered the same issue, while scraping one directory.

Here's the forum link, so that you can make some sense of it: http://www.ubotstudio.com/forum/index.php?/topic/16788-csv-based-ecommerce-data-extraction-bot-product-description-product-picture-scraping/

 

Maybe we can work together in finding a solution?

Link to post
Share on other sites

Just FYI, you are commenting on 4 years old topic, and OP was using different UBot version...v3...so I doubt his and yours problems are related...case 

 

In your case I would open a new topic or contact the support.

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...