josh12345 0 Posted October 1, 2010 Report Share Posted October 1, 2010 I have a simple bot I made that scrapes for two pieces of data from every url I add to a list for ubot to navigate to. I'm actually having two issues with this little bot. The first (and my initial reason for posting this) is that every 20th or so url ubot navigates to, it gets stuck on that url and won't continue on to the other thousand or so pages I want it to scrape. Second, it's basically a workaround of what I really want to acclomplish with this tool. One of the pieces of data I'm trying to scrape shows up maybe 50 to 75% of the time in the meta decription of these pages, so I've been settling for scraping that instead of what I really want to scrape on the page. The second thing I'm scraping is a specific url off each page, but I can't figure out how to get the specific one I want so I'm scraping them all which is usually about 10 extras per page and then I (sort of) filter them out. I'm uploading a screenshot of what I have so far with my ubot code and an example of the two pieces of data I'm scraping, so if anyone has any suggestions on any of these issues that would be great. Thanks! First half of code: Second half: Example of pages I'm scrape and what I'm scraping: Quote Link to post Share on other sites
meter 145 Posted October 1, 2010 Report Share Posted October 1, 2010 Try not executing the code from the second half. Check if the bot lasts longer this way. Just want to check a suspicion. Quote Link to post Share on other sites
josh12345 0 Posted October 1, 2010 Author Report Share Posted October 1, 2010 Meter, could you be more specific? Should I just try to scrape the meta description and leave off scaping the urls and see if that is what caused the problem? Quote Link to post Share on other sites
meter 145 Posted October 1, 2010 Report Share Posted October 1, 2010 Yep. Quote Link to post Share on other sites
josh12345 0 Posted October 3, 2010 Author Report Share Posted October 3, 2010 Ok, so I tried to run it the way meter suggested, scraping just the meta description. I ran the bot and tracked it's results, and it got stuck on the 55th url. I think it's made it that far before. I'm still looking for a solution for this issue, so if anybody has any other suggestions about how to solve this problem that would be a big help. I know this software is capable of doing tasks like this without freezing up, because I made a different simple bot that's been running for the last 2 weeks or so non stop without freezing! Quote Link to post Share on other sites
Askabar 1 Posted September 9, 2014 Report Share Posted September 9, 2014 Hey Josh, Wanted to let you know, that I encountered the same issue, while scraping one directory.Here's the forum link, so that you can make some sense of it: http://www.ubotstudio.com/forum/index.php?/topic/16788-csv-based-ecommerce-data-extraction-bot-product-description-product-picture-scraping/ Maybe we can work together in finding a solution? Quote Link to post Share on other sites
UBotDev 276 Posted September 9, 2014 Report Share Posted September 9, 2014 Just FYI, you are commenting on 4 years old topic, and OP was using different UBot version...v3...so I doubt his and yours problems are related...case In your case I would open a new topic or contact the support. Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.