Multi-Thread & HTTP GET

[Brutal 164]

Google tends to frown upon hitting them to quickly when you are using http get to scrape code (well, for that matter they don't like it when you use browser either - but the ban seems to come faster with http get).

 

Anyway - I was just wondering if you hit them with all of the requests at the exact same time, if that would get you flagged just as quickly, or if because they hit all at once, it's fast enough to skip the trigger for that run.

 

So I was thinking about doing a multi-thread just to grab the code from the target page for all pages at once.

 

It's a total of 40 pages ... So open 40 threads all at once to grab the code.

 

Anyone tried anything similar - If so, was the results still the same ban-hammer, or did you get by with it?

[Bot-Factory 602]

Google tends to frown upon hitting them to quickly when you are using http get to scrape code (well, for that matter they don't like it when you use browser either - but the ban seems to come faster with http get).

 

Anyway - I was just wondering if you hit them with all of the requests at the exact same time, if that would get you flagged just as quickly, or if because they hit all at once, it's fast enough to skip the trigger for that run.

 

So I was thinking about doing a multi-thread just to grab the code from the target page for all pages at once.

 

It's a total of 40 pages ... So open 40 threads all at once to grab the code.

 

Anyone tried anything similar - If so, was the results still the same ban-hammer, or did you get by with it?

 

That doesn't work. The technology is intelligent enough to detect stuff like that. Requests are never coming in at the same time. You might send the TCP packages within multiple threads. 

But you still have a single internet connection where all the packages are processed sequentially. 

 

So, a typical TCP/IP host has multiple processes each needing to send and receive datagrams. All of them, however, must be sent using the same interface to the network, using the IP layer. This means that the data from all applications is “funneled down”, initially to the transport layer, where it is handled by either TCP or UDP. From there, messages pass to the device's IP layer, where they are packaged in IP datagrams and sent out over the internet to different destinations. The technical term for this is multiplexing. This term simply means combining, and its use here is a software analog to the way it is done with signals.

 

So that way it can still be detected.

 

Proxies are the only way to work around the issue you described. Well.. solving the google captcha might work as well to some degree.

 

Dan

 

 

[Brutal 164]

Holy crap Dan! - Ha! Your brain is firing on all 8 cylinders man! Thanks for the details explanation.

[drhum 3]

Really nice. I'm a visual person and seeing that diagram is good enough already! Thanks.