Jump to content
UBot Underground

Ubot Error : Cannot Deserialize The Current Json...


Recommended Posts

Bealad00,

Yes, I did see that error yesterday actually. I was changing the value of a column within a loop that was the subject of comparison in the loop while command.

I changed that column used in loop while to another variable. Once done with changing the original variable inside the loop I set the new variable used in the loop while comparison.

Sorry if the answer is confusing.

I think there was a timing issue.

That seemed to solve that issue above but eventually I moved the whole project back to UBOT version 4 from 5.5 because of other issues.

Link to post
Share on other sites

HAha i got the same thing on my error and nobody could help me with this error even Ubot support had no ideawhat to say to this!

 

i was scraping Information from a site and i set table cells and after 18th loop this started happening,all i did is set user agent to internet explorer

Cleared cookies and i didn't allowed any flash or css on the website and this seemed to help!

 

i am sharing my experience because it was frustrating that nobody had no idea how to help me i hope my answer will be helpful!

Link to post
Share on other sites
  • 2 months later...

Ok, I am going to jump into this thread. I am having the same problem. 

 

I have tried increasing the wait time between page reloads in my scrapper (just make sure everything was fully loaded and account for sluggish internet connections).  I get about 100 pages into a scrape and I get the same messages.

 

I am working on a theory.  Could Google Adsense on a site be a problem?  After all, scripts like these could throw off their impression metrics (something they pride themselves on).  Just not sure how to test the idea?  How can we do ad blocking?

Link to post
Share on other sites

Here is the Define it's happening in.  I had this chuck of code nested before.  I thought breaking it out might help me see the problem better.

 

define Get Identity {
    set(#Identity,$scrape attribute(<class="info">,"innerhtml"),"Global")
    set(#Identity,$replace regular expression(#Identity,"\\t|\\r|\\n| ",$nothing),"Global")
    set(#Identity,$replace regular expression(#Identity,"(?<=>)\\s+(?=<)",$nothing),"Global")
    set(#Identity,$replace regular expression(#Identity,"(?<=>)\\s+(?=[a-zA-Z0-9\\:])",$nothing),"Global")
    set(#Identity,$replace regular expression(#Identity,"(?<=[a-zA-Z0-9\\:\\?])\\s+(?=<)",$nothing),"Global")
}
 

It's a basic scrape and cleanup of the HTML.

 

This is the only scrape I am doing on each page as it comes up.  From there I just pick out data and add it to lists... nothing special. It basically crashes on the first Set command after about 100 pages or so.

 

Not sure if a Ubot gets any more basic than this.

Link to post
Share on other sites

Here is an additional error message I am getting after the JSON.

 

Error converting value True to type 'System.Collections.Generic.List`1[system.String]'.Path ", line 1, position 4.

Source: pagescraperbot -> -> set -> $scrape attribute (<class="info">, "innerhtml")-> $scrape attribute(<class="info">,innerhtml")->$scrape attribute

Link to post
Share on other sites

Ugh... OK, I moved just the working/running parts of my bot to a whole new file.  (I had some other scripts/tabs in the original file that were still in development).  Ran a test to loop 300 pages and it worked without error. I will test on a higher number next.  I will post back here if there is still a problem. Otherwise, no news is good news.  Thanks.

Link to post
Share on other sites

sounds like the scrape attribute is being fired before the page loads,the replace nodes,wouldn't cause an error like that,they would just return nothing,so the problem I reckon is with the scrape attribute

 

loop while($comparison($exists(<class="info">),"=","false")) {
    wait(0.2)
}

set(#Identity,"","Global")
set(#Identity,$scrape attribute(<class="info">,"innerhtml"),"Global")
set(#Identity,$replace regular expression(#Identity,"\\t|\\r|\\n| ",$nothing),"Global")
set(#Identity,$replace regular expression(#Identity,"(?<=>)\\s+(?=<)",$nothing),"Global")
set(#Identity,$replace regular expression(#Identity,"(?<=>)\\s+(?=[a-zA-Z0-9\\:])",$nothing),"Global")
set(#Identity,$replace regular expression(#Identity,"(?<=[a-zA-Z0-9\\:\\?])\\s+(?=<)",$nothing),"Global")
 

  • Like 1
Link to post
Share on other sites

Thanks for the suggestion and great support.  I thought the loop would work but once again I get an error... 

 

http://i.imgur.com/UEfhNtL.jpg?1

 

Then this JSON error after hitting "Continue".

 

http://i.imgur.com/b7qGD0Q.jpg

 

I will try your $exists loop on the <id=genbtn"> html also...  Hope that makes a difference.

Link to post
Share on other sites

I have tried several things to address this problem in addition to the above.  First, removing plugins that weren't needed in the bot.  Second uninstalling recently added programs on my PC. I also disabled firewalls and security software. Finally, I uninstalled and reinstalled UbotStudio.  I am still getting the same errors.  Anyone else having these issues when scraping over 1000 pages? (mine will error out somewhere around 300 page +/-)

Link to post
Share on other sites

yes I started playing with the scrape attribute node and I started getting this error,why not try $document text and regex it,to match multiple lines use (?s)

  • Like 1
Link to post
Share on other sites

Thanks for the suggestion. I gave the $document text a try.  Got similar errors.  The first one is different and says

 

error converting value "<html.... (the rest the result $document text html follows)

 

So I thought I would make a loop that tested the validity of the $document text just to make sure it was grabbing the HTML.  Basically I would wait 4  seconds to make sure the page loads, grab the document, wait another 2 seconds just to slow it down, test if it was holding html, if not try again.

 

That failed too.

 

These errors feel like a memory leak to me... the bugs are impacting my ROI.

Link to post
Share on other sites

so what did the document text say when it failed? nothing? actually,do you have an $eval node anywhere in your script? if so remove it,I don't know much javascript and I started playing with that and I got some VERY unexpected results,that was one of the problems for me

  • Like 1
Link to post
Share on other sites

This was the error from the $document text.

 

error converting value "<html.... (the rest the result $document text html follows)

 

No $eval nodes.  I am really just grabbing HTML and picking it apart with regular expressions, storing data in lists for writing out. If the logic were bad it would fail on the first few times (I would think) and not wait till a few hundred loops into the script. 

 

It's almost like the browser is crashing or something?

 

Could the target site be blocking me?

Link to post
Share on other sites

Did you notice your browser is crashing?

 

Try to Open every new link from your List into new browser

(I assume whatever you are scraping you first Scrape URL-s to collect all URL-s from some page and then 

you probably navigate to every page using $list item or $next list item

 

So in your Loop set it up so every time Script navigates to the URL from $list item

 

so here is the idea behind it

 

set #list position = 0

loop {

     Open new browser

     Scrape data

     close new browser

     increment #list position

}

 

This way your Browser will always be Fresh and Your script wont crash i am using the same logic with my Software and it works Flawlessly for my purposes,i just thought if this was a Browser issue maybe you could try it.

 

Best Regards!

  • Like 2
Link to post
Share on other sites

itexspert - Good idea.  I will have to give it some thought.  The data I am scraping is generated dynamically on the page via button click and on page parameters so there isn't a url list to generate. It's doable I will just have to reset the parameters for each new browser... I will probably just get duplicate data.  I will let you know.

Link to post
Share on other sites

Having some success. I scraped 1000 pages last night.

 

The bot was failing at about 300 loops.  Still don't know why.  But I added an inner loop to wait 3 minutes every 100 iterations. Slowing it down seem to help. I am testing it now on a larger count.

Link to post
Share on other sites
  • 2 weeks later...

OK I am back - I tried compiling the bot and running it that way.  That failed even worse. So, I added a number of error handling routines along different places in the bot to try to capture the error and correct for possible errors.  While I am able now to get up to 2000 iterations/pages it still locks up.  Am I asking too much from Ubot Studio? Is it meant for bigger data collections? Don't take this wrong, I love the product and the support but I often feel like I am a beta tester and not a user.

Link to post
Share on other sites
  • 3 months later...

I have exactly the same errors when trying to open Facebook pages and scroll them down and down again to scrape information about the posts published on that Facebook page. When and where it happens is a kind of random: sometimes it scrolls down normally, but sometimes (when the page is growing bigger than cca. 11 MBs), then it throws errors like the ones which are mentioned above. 

 

What should I do to prevent this from happening? (currently using the latest version, that is 5.7.1)

Link to post
Share on other sites
  • 2 months later...

I get this error on many sites.  I am experiencing it on flippa now, but also many others.  I published several trouble tickets on this - ubot always says they cannot reproduce the problem.

Link to post
Share on other sites

There are mainly 2 reasons for this issue:

 

1. The awesomnium browser engine runs into some memory limitations. Or some other error.

2. The website you try to automate is able to detect bots. Maybe you scroll to fast. You don't move naturally. There are lots of techniques out there to detect human behavior and to figure out if a human is navigating and using the site, or a machine. And I'm not talking about captchas. I talk about same timeouts when scrolling, mouse movements and so on.

 

Kindest regards

Dan

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...