itexspert 47 Posted November 17, 2014 Report Share Posted November 17, 2014 Guys i need some help with this you see i am trying to build something i am stuck here http://www.gramfeed.com/instagram/tags#models I am trying to click on these smaller circled images because these take you to the Profile of the user i tried to scrape it and put it in a list but no luck i attached a picture so you can see which info exactly i need to click to , or save links in a list whatever you think is best .Because i am trying to make Ubot loop trough these links using either positions or something i am trying to do this for like 2 days now no luck,if someone could help i would appreciate it. Example 2http://www.bluemountainpeak.com/catalogsearch/result/?q=00 So here i am trying to click on every product Individually then scrape some info go back to the site and choose next product on the webpage. I would be grateful if you could take a look at these two i am stuck with this! Thanks Guys! Quote Link to post Share on other sites
HaHaItsJake 25 Posted November 17, 2014 Report Share Posted November 17, 2014 Hey itexspert, Let me know if this helps you out. For some reason, gramfeed wouldn't do anything for me via Ubot browser. Wouldn't click anything. not sure if it was just me, but here what I came up with for both sites. (Same Method) Couldn't get clicking to work at all. http://www.gramfeed.com/instagram/tags#models: clear cookies set user agent("Chrome") navigate("http://www.gramfeed.com/instagram/tags#models", "Wait") set(#Profile_URL, $scrape attribute(<class="photo_user">, "title"), "Global") add list to list(%Profile_URL, $list from text(#Profile_URL, $new line), "Delete", "Global") loop($list total(%Profile_URL)) { navigate("http://www.gramfeed.com/{$next list item(%Profile_URL)}", "Wait") wait(10) } http://www.bluemountainpeak.com/catalogsearch/result/?q=00: clear cookies set user agent("Chrome") navigate("http://www.bluemountainpeak.com/catalogsearch/result/?q=00", "Wait") set(#Product_Url, $scrape attribute(<div,class="details-wrap">, "innerhtml"), "Global") set(#Product_Url, $replace($find regular expression(#Product_Url, "(?<=\\<a href\\=\\\").*?(?<=\\\" title)"), "\" title", ""), "Global") add list to list(%Product_Url, $list from text(#Product_Url, $new line), "Delete", "Global") loop($list total(%Profile_URL)) { navigate($next list item(%Profile_URL), "Wait") wait(10) } Let me know if it's helps you or not. Regards,HaHaItsJake Quote Link to post Share on other sites
itexspert 47 Posted November 17, 2014 Author Report Share Posted November 17, 2014 Thanks i will test it out tell me does Browser change settings when you switch it to Chrome? Quote Link to post Share on other sites
itexspert 47 Posted November 17, 2014 Author Report Share Posted November 17, 2014 One more thing is there a way to limit how many URL-s will be visited from the list clear cookiesset user agent("Chrome")navigate("http://www.gramfeed.com/instagram/tags#models", "Wait")set(#Profile_URL, $scrape attribute(<class="photo_user">, "title"), "Global")add list to list(%Profile_URL, $list from text(#Profile_URL, $new line), "Delete", "Global")loop($list total(%Profile_URL)) {<<<<<<<<<Is there a way to add Prompt Command here so that User can limit how many users he wants to visit? or add a Ui textbox with variable in which User can just write how many users he wants to visit from the list is that possible? navigate("http://www.gramfeed.com/{$next list item(%Profile_URL)}", "Wait")wait(10)} Quote Link to post Share on other sites
HaHaItsJake 25 Posted November 17, 2014 Report Share Posted November 17, 2014 (edited) What settings do you mean? Chrome is just the user agent that their websites picks up. You can change that to what ever. Setting it to mobile is smart for some websites. For the following code, theirs 2 ways of doing it. In the UI, or a Dialog box. (I can't remember for the life of me what the command was. I can't find the script I have it in either. I believe it's a plugin...)But the UI would be simple, ui text box("How mNay URLS", #Urls) loop(#Urls) { } Regards,HaHaItsJakeEDIT: Found the command. Duhhh. Yeah it will be: loop($prompt("How many urls")) { } Or add the $prompt to a $set. Edited November 17, 2014 by HaHaItsJake 1 Quote Link to post Share on other sites
itexspert 47 Posted November 17, 2014 Author Report Share Posted November 17, 2014 Thanks man you have been very helpful! Quote Link to post Share on other sites
itexspert 47 Posted November 19, 2014 Author Report Share Posted November 19, 2014 Hey guys try to open this site in your Ubot http://www.ripoffreport.com/r/czar-delivery-usa-inc-/coral-springs-florida-33076/car-delivery-usa-inc-all-day-auto-transport-fraudulent-bait-and-switch-said-800-at-deli-1184929 I am trying to scrape address and Phone number,Web,and web address but my Ubot keeps shutting down???? Can you guys try is this happening to you? Quote Link to post Share on other sites
HaHaItsJake 25 Posted November 19, 2014 Report Share Posted November 19, 2014 Hey guys try to open this site in your Ubot http://www.ripoffreport.com/r/czar-delivery-usa-inc-/coral-springs-florida-33076/car-delivery-usa-inc-all-day-auto-transport-fraudulent-bait-and-switch-said-800-at-deli-1184929 I am trying to scrape address and Phone number,Web,and web address but my Ubot keeps shutting down???? Can you guys try is this happening to you?Loading perfectly in Ubot 4. Can play around with it too. Try the following browser settings after saving all your work and re-opening Ubot Complete (Maybe a Reboot too?) The numbers and everything still come up but load %80 faster for me. set referrer("google.com") set user agent("Chrome") allow flash("No") allow javascript("No") allow popups("No") allow css("No") allow images("No") navigate("http://www.ripoffreport.com/r/czar-delivery-usa-inc-/coral-springs-florida-33076/car-delivery-usa-inc-all-day-auto-transport-fraudulent-bait-and-switch-said-800-at-deli-1184929", "Wait") Regards,HaHaItsJake 1 Quote Link to post Share on other sites
itexspert 47 Posted November 19, 2014 Author Report Share Posted November 19, 2014 Ok it worked temporarily this is the new issue clear cookiesset user agent("Android")allow flash("No")allow javascript("No")allow popups("No")allow css("No")allow images("No")increment(#row)clear list(%ripoff)allow images("No")navigate("http://www.ripoffreport.com/c/483/automotive/auto-mechanics", "Wait")wait(10)set(#ripoofreport, $scrape attribute(<href=w"http://www.ripoffreport.com/r/*">, "href"), "Global")add list to list(%ripoff, $list from text(#ripoofreport, $new line), "Delete", "Global")loop($list total(%ripoff)) { navigate($next list item(%ripoff), "Wait") clear cookies set user agent("Android") wait(10) set(#adresaripoff, $scrape attribute(<class="address">, "innertext"), "Global") add item to list(%Scraped Results, $list from text(#adresaripoff, $new line), "Delete", "Global") set(#inforipoff, $scrape attribute($element offset(<tagname="ul">, 13), "innertext"), "Global") add list to table as column(&ripoffreport, #row, 0, %Scraped Results) set table cell(&ripoffreport, #row, 1, #inforipoff)} Idea behind this is that we go to some category from this site ripoffthen Scrape all links in this category we are input it in the Listand visit them one by one and scrape info i need Problem is Ubot V4 and V5 are stuck on loading pageremember that list we created with links which leads you directly to the page from which i need to scrape infowell after Ubot goes to this link it gets stuck on loading browser cant seem to load the content from the page?? Why is this happening i am trying but no luck with both versions! Quote Link to post Share on other sites
HaHaItsJake 25 Posted November 19, 2014 Report Share Posted November 19, 2014 You got some glitches in there. To learn, compare your code with mine. I also added comments along the way. I didn't run it, so the scrape might be right. Having a Element offset on more then 1 link will get messed up. clear cookies set user agent("Android") allow flash("No") allow javascript("No") allow popups("No") allow css("No") allow images("No") comment("Need to set the row to 0. Can't Increment because it doesn't have a value. That in return will screw up the Table Cells at the end of the loop.") set(#row, 0, "Global") clear list(%ripoff) comment("Need to clear the table, or it will be added on.") clear table(&ripoffreport) comment("Had an extra allow images. I deleted it. Not need for it.") navigate("http://www.ripoffrep.../auto-mechanics", "Wait") wait(10) set(#ripoofreport, $scrape attribute(<href=w"http://www.ripoffreport.com/r/*">, "href"), "Global") add list to list(%ripoff, $list from text(#ripoofreport, $new line), "Delete", "Global") loop($list total(%ripoff)) { navigate($next list item(%ripoff), "Wait") comment("The reason why it's not loading is the fact that you go to the page, then it clears the cookies and sets the user agent. It's already set and wouldn't clear the cookies unless you're using proxies.") wait(10) set(#adresaripoff, $scrape attribute(<class="address">, "innertext"), "Global") add item to list(%Scraped Results, $list from text(#adresaripoff, $new line), "Delete", "Global") comment("You need to fix the inforripoff scrape. It's nopt scraping after the first URL.") set(#inforipoff, $scrape attribute($element offset(<tagname="ul">, 13), "innertext"), "Global") add list to table as column(&ripoffreport, #row, 0, %Scraped Results) set table cell(&ripoffreport, #row, 1, #inforipoff) comment("For the Row on the Table cells, it will replace the whole table each loop. I would either move the Add List To Table outside the loop. Or find a way to get the list total and increamnt it. Or Table Row Total for the Row.") } Regards,HaHaItsJake 1 Quote Link to post Share on other sites
HaHaItsJake 25 Posted November 19, 2014 Report Share Posted November 19, 2014 (edited) 1 second, that code got messed up some how. EDIT: Code Fixed Above. Edited November 19, 2014 by HaHaItsJake Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.