15 replies to this topic

[mdc101]

Hi Guys

I have created a large list of user agents
The one file scraped by my bot was about 8900.
I then ran a test to work on swapping the user agent

ui open file("Mobile useragents: ", #MobileUserAgentList)
ui open file("Browser user agents:", #BrowserUserAgentList)
add list to list(%CurrentUserAgentList, $list from text(#BrowserUserAgentList, $new line), "Delete", "Global")
set user agent($random list item(%CurrentUserAgentList))
navigate("http://show-ip.net/browserinfo/", "Wait")

As soon as we get to
set user agent($random list item(%CurrentUserAgentList))

The bot seems to take ages - it does not hang but just does not proceed.

I wanted to see how quick the random functions works on large lists.

Is there something wrong with the code or cant ubot handle such large lists?

I have attached the user agent files if anyone is looking for user agent lists.

Thanks for any feedback

Attached Files

•  mobile-user-agent-list.txt   40.85K   23 downloads
•  user-agent-list.txt   791.17K   38 downloads

[Duane]

Have you tried it with a smaller file to see if the code is sound?

[Bob The Builder]

Hi Guys

I have created a large list of user agents
The one file scraped by my bot was about 8900.
I then ran a test to work on swapping the user agent

ui open file("Mobile useragents: ", #MobileUserAgentList)
ui open file("Browser user agents:", #BrowserUserAgentList)
add list to list(%CurrentUserAgentList, $list from text(#BrowserUserAgentList, $new line), "Delete", "Global")
set user agent($random list item(%CurrentUserAgentList))
navigate("http://show-ip.net/browserinfo/", "Wait")

As soon as we get to
set user agent($random list item(%CurrentUserAgentList))

The bot seems to take ages - it does not hang but just does not proceed.

I wanted to see how quick the random functions works on large lists.

Is there something wrong with the code or cant ubot handle such large lists?

I have attached the user agent files if anyone is looking for user agent lists.

Thanks for any feedback

do a search I made a snippet that works well for this but much smaller list.

[JohnB]

Try list from file instead of list from text...

John

[mdc101]

HI
I have shrunk the file down to about 27 lines and it runs through pretty quickly.
What I did notice is it changes the useragent every 5th run.

Is this how the random list item works?
Is there a way to say grab 25 random urls from a large list text file and then have the used as the useragents and on every 5th run it grabs a new set of say 25 random urls?

Ideally one would think that random list item would be every run.

[JohnB]

Ok, the problem is the size of the list...if you watch this, I am using the whole list and the switch useragent is where it gets stuck:

http://screencast.com/t/I0Ig3Hsae

When I switched it to only use 300 of the list items it works fine:

http://screencast.com/t/hmcnXv4EH3

You will notice my setup in the video does not use random. I set up my own random function simply so I could control the size of the list without changing it. The entire list is loaded, but I am only using the first 300 items in the second video. It works just as well with $random list item. (which is the correct way to do it)

John

clear list(%useragents)
add list to list(%useragents, $list from file("{$special folder("Desktop")}\\user-agent-list.txt"), "Delete", "Global")
loop(25) {
set(#position, $rand(0, 299), "Global")
set user agent($list item(%useragents, #position))
navigate("http://whatsmyuseragent.com/", "Wait")
wait(4)
}

[mdc101]

Ah very clever! Thanks John that's a cool way of working with a large list where you need randomness.

Thanks guys for the feedback, appreciated

[Alliance]

Hi I am new in this forum, and also in the Ubot world.

I am setting my first bot and discovered this ppost to be usefull for me, but unfortunatelly I can't download the useragent list attached.

Is there an special precedure to download files in this forum, it says forbitten acces for me, but I am logged in the forum.

Thanks for your help.

[toothless]

Hi I am new in this forum, and also in the Ubot world.

I am setting my first bot and discovered this ppost to be usefull for me, but unfortunatelly I can't download the useragent list attached.

Is there an special precedure to download files in this forum, it says forbitten acces for me, but I am logged in the forum.

Thanks for your help.

I have the same problem,What can I do also,very nice post

[a2mateit]

I have the same problem,What can I do also,very nice post

You need to PM a moderator to get your restriction lifted...

[Website Rich]

Why are all the attachments down on every post I visit? How do we download the ubots and txt files if they are all missing? Or is there a level you need to be before you're able to download any files? Please advise me on what I need to do. Thanks

[Duane]

Why are all the attachments down on every post I visit? How do we download the ubots and txt files if they are all missing? Or is there a level you need to be before you're able to download any files? Please advise me on what I need to do. Thanks

Are you registered for this forum? If not just send in a support ticket to axx the goodies...

[Twitter Dominator]

im registered and their not loading for me either, but anyways the code is pasted for a copy paste in a few posts above so you dont need the lists to make it work you can scrape the useragent list from almost everywhere now, the code works in the abvove post
http://www.ubotstudi...nts/#entry44072

[rodvaN]

This is a good topic, too bad the user agent list he uploaded is not up anymore.
Anything we can do?

[Traffik Cop]

Anyone have this list to re upload for us? would like to have!!

[AutomationNinja]

user_pref("useragentswitcher.1.appname", "");
user_pref("useragentswitcher.1.appversion", "");
user_pref("useragentswitcher.1.description", ":: BROWSERS - Windows ::");
user_pref("useragentswitcher.1.useragent", "");
user_pref("useragentswitcher.10.description", "Multizilla 1.6 (Win xp)");
user_pref("useragentswitcher.10.useragent", "Mozilla/5.0 (Windows; U; Windows XP) Gecko MultiZilla/1.6.1.0a");
user_pref("useragentswitcher.11.description", "Netscape 7.1 (Win 98)");
user_pref("useragentswitcher.11.useragent", "Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.4) Gecko Netscape/7.1 (ax)");
user_pref("useragentswitcher.12.description", "Netscape 4.8 (Win XP)");
user_pref("useragentswitcher.12.useragent", "Mozilla/4.8 [en] (Windows NT 5.1; U)");
user_pref("useragentswitcher.13.description", "Netscape 3.01 gold (Win 95)");
user_pref("useragentswitcher.13.useragent", "Mozilla/3.01Gold (Win95; I)");
user_pref("useragentswitcher.14.description", "Netscape 2.02 (Win 95)");
user_pref("useragentswitcher.14.useragent", "Mozilla/2.02E (Win95; U)");
user_pref("useragentswitcher.15.description", ":: SPIDERS - search ::");
user_pref("useragentswitcher.15.useragent", "");
user_pref("useragentswitcher.16.description", "Googlebot 2.1 (New version)");
user_pref("useragentswitcher.16.useragent", "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)");
user_pref("useragentswitcher.17.description", "Googlebot 2.1 (Older Version)");
user_pref("useragentswitcher.17.useragent", "Googlebot/2.1 (+http://www.googlebot.com/bot.html)");
user_pref("useragentswitcher.18.description", "Msnbot 1.0 (current version)");
user_pref("useragentswitcher.18.useragent", "msnbot/1.0 (+http://search.msn.com/msnbot.htm)");
user_pref("useragentswitcher.19.description", "Msnbot 0.11 (beta version)");
user_pref("useragentswitcher.19.useragent", "msnbot/0.11 (+http://search.msn.com/msnbot.htm)");
user_pref("useragentswitcher.2.appname", "Microsoft Internet Explorer");
user_pref("useragentswitcher.2.appversion", "4.0 (Compatible; MSIE 6.0; Windows NT 5.1)");
user_pref("useragentswitcher.2.description", "MSIE 6 (Win XP)");
user_pref("useragentswitcher.2.useragent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
user_pref("useragentswitcher.20.description", "Yahoo Slurp");
user_pref("useragentswitcher.20.useragent", "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.co...s/ysearch/slurp)");
user_pref("useragentswitcher.21.description", "Ask Jeeves/Teoma");
user_pref("useragentswitcher.21.useragent", "Mozilla/2.0 (compatible; Ask Jeeves/Teoma)");
user_pref("useragentswitcher.22.description", ":: BROWSERS -  Mac ::");
user_pref("useragentswitcher.22.useragent", "");
user_pref("useragentswitcher.23.description", "Safari 125.8 (Mac OSX)");
user_pref("useragentswitcher.23.useragent", "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/125.2 (KHTML, like Gecko) Safari/125.8");
user_pref("useragentswitcher.24.description", "Safari 85 (Mac OSX)");
user_pref("useragentswitcher.24.useragent", "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/125.2 (KHTML, like Gecko) Safari/85.8");
user_pref("useragentswitcher.25.description", "MSIE 5.15 (Mac OS 9)");
user_pref("useragentswitcher.25.useragent", "Mozilla/4.0 (compatible; MSIE 5.15; Mac_PowerPC)");
user_pref("useragentswitcher.26.description", "Firefox 0.9 (Mac OSX )");
user_pref("useragentswitcher.26.useragent", "Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7a) Gecko/20040614 Firefox/0.9.0+");
user_pref("useragentswitcher.27.description", "Omniweb563 (Mac OSX)");
user_pref("useragentswitcher.27.useragent", "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-US) AppleWebKit/125.4 (KHTML, like Gecko, Safari) OmniWeb/v563.15");
user_pref("useragentswitcher.28.description", ":: BROWSERS - linux/unix/beos ::");
user_pref("useragentswitcher.28.useragent", "");
user_pref("useragentswitcher.29.description", "Epiphany 1.2 (Linux)");
user_pref("useragentswitcher.29.useragent", "ozilla/5.0 (X11; U; Linux; i686; en-US; rv:1.6) Gecko Epiphany/1.2.5");
user_pref("useragentswitcher.3.appname", "Microsoft Internet Explorer");
user_pref("useragentswitcher.3.appversion", "4.0 (compatible; MSIE 5.5; Windows NT 5.0)");
user_pref("useragentswitcher.3.description", "MSIE 5.5 (Win 2000)");
user_pref("useragentswitcher.3.useragent", "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0 )");
user_pref("useragentswitcher.30.description", "Epiphany 1.4 (Ubuntu)");
user_pref("useragentswitcher.30.useragent", "Mozilla/5.0 (X11; U; Linux i586; en-US; rv:1.7.3) Gecko/20040924 Epiphany/1.4.4 (Ubuntu)");
user_pref("useragentswitcher.31.description", "Firefox 0.8 (Linux)");
user_pref("useragentswitcher.31.useragent", "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040614 Firefox/0.8");
user_pref("useragentswitcher.32.description", "Galeon 1.3 (Linux)");
user_pref("useragentswitcher.32.useragent", "Mozilla/5.0 (X11; U; Linux; i686; en-US; rv:1.6) Gecko Galeon/1.3.14");
user_pref("useragentswitcher.33.description", "Konqueror 3 rc4 (Linux)");
user_pref("useragentswitcher.33.useragent", "Konqueror/3.0-rc4; (Konqueror/3.0-rc4; i686 Linux;;datecode)");
user_pref("useragentswitcher.34.description", "Konqueror 3.3  (Gentoo)");
user_pref("useragentswitcher.34.useragent", "Mozilla/5.0 (compatible; Konqueror/3.3; Linux 2.6.8-gentoo-r3; X11;");
user_pref("useragentswitcher.35.description", "Mozilla 1.6 (Debian)");
user_pref("useragentswitcher.35.useragent", "Mozilla/5.0 (X11; U; Linux; i686; en-US; rv:1.6) Gecko Debian/1.6-7");
user_pref("useragentswitcher.36.description", "Opera 7.23 (Linux)");
user_pref("useragentswitcher.36.useragent", "MSIE (MSIE 6.0; X11; Linux; i686) Opera 7.23");
user_pref("useragentswitcher.37.description", "ELinks 0.9.3 (Kanotix)");
user_pref("useragentswitcher.37.useragent", "ELinks/0.9.3 (textmode; Linux 2.6.9-kanotix-8 i686; 127x41)");
user_pref("useragentswitcher.38.description", "Elinks 0.4pre5 (Linux)");
user_pref("useragentswitcher.38.useragent", "ELinks (0.4pre5; Linux 2.6.10-ac7 i686; 80x33)");
user_pref("useragentswitcher.39.description", "Links 2.1 (Linux)");
user_pref("useragentswitcher.39.useragent", "Links (2.1pre15; Linux 2.4.26 i686; 158x61)");
user_pref("useragentswitcher.4.description", "MSIE 5.5 (Win ME)");
user_pref("useragentswitcher.4.useragent", "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)");
user_pref("useragentswitcher.40.description", "Links 0.9.1 (Linux)");
user_pref("useragentswitcher.40.useragent", "Links/0.9.1 (Linux 2.4.24; i386;)");
user_pref("useragentswitcher.41.description", "Lynx 2.8.5rel.1 (Linux)");
user_pref("useragentswitcher.41.useragent", "Lynx/2.8.5rel.1 libwww-FM/2.14 SSL-MM/1.4.1 GNUTLS/0.8.12");
user_pref("useragentswitcher.42.description", "w3m 0.5.1 (Linux)");
user_pref("useragentswitcher.42.useragent", "w3m/0.5.1");
user_pref("useragentswitcher.43.description", "Links 2.1 (FreeBSD)");
user_pref("useragentswitcher.43.useragent", "Links (2.1pre15; FreeBSD 5.3-RELEASE i386; 196x84)");
user_pref("useragentswitcher.44.description", "Mozilla 1.7 (FreeBSD)");
user_pref("useragentswitcher.44.useragent", "Mozilla/5.0 (X11; U; FreeBSD; i386; en-US; rv:1.7) Gecko");
user_pref("useragentswitcher.45.description", "Netscape 4.77 (Irix)");
user_pref("useragentswitcher.45.useragent", "Mozilla/4.77 [en] (X11; I; IRIX;64 6.5 IP30)");
user_pref("useragentswitcher.46.description", "Netscape 4.8 (SunOS)");
user_pref("useragentswitcher.46.useragent", "Mozilla/4.8 [en] (X11; U; SunOS; 5.7 sun4u)");
user_pref("useragentswitcher.47.description", "Net Positive 2.1 (BeOS)");
user_pref("useragentswitcher.47.useragent", "Mozilla/3.0 (compatible; NetPositive/2.1.1; BeOS)");
user_pref("useragentswitcher.48.description", ":: SPIDERS - misc ::");
user_pref("useragentswitcher.48.useragent", "");
user_pref("useragentswitcher.49.description", "download demon");
user_pref("useragentswitcher.49.useragent", "Download Demon/3.5.0.11");
user_pref("useragentswitcher.5.appname", "");
user_pref("useragentswitcher.5.appversion", "");
user_pref("useragentswitcher.5.description", "Avant Browser 1.2");
user_pref("useragentswitcher.5.useragent", "Avant Browser/1.2.789rel1 (http://www.avantbrowser.com)");
user_pref("useragentswitcher.50.description", "Email Wolf");
user_pref("useragentswitcher.50.useragent", "EmailWolf 1.00");
user_pref("useragentswitcher.51.description", "grub client");
user_pref("useragentswitcher.51.useragent", "grub-client-1.5.3; (grub-client-1.5.3; Crawl your own stuff with http://grub.org)");
user_pref("useragentswitcher.52.description", "gulperbot");
user_pref("useragentswitcher.52.useragent", "Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)");
user_pref("useragentswitcher.53.description", "ms url control");
user_pref("useragentswitcher.53.useragent", "Microsoft URL Control - 6.00.8862");
user_pref("useragentswitcher.54.description", "omni web");
user_pref("useragentswitcher.54.useragent", "OmniWeb/2.7-beta-3 OWF/1.0");
user_pref("useragentswitcher.55.description", "winHTTP");
user_pref("useragentswitcher.55.useragent", "SearchExpress");
user_pref("useragentswitcher.56.description", ":: v: 1.0.4 :: techpatterns.com ::");
user_pref("useragentswitcher.56.useragent", "");
user_pref("useragentswitcher.56.vendor", "http://techpatterns.com/forums/about304.html");
user_pref("useragentswitcher.6.appname", "Microsoft Internet Explorer");
user_pref("useragentswitcher.6.appversion", "4.0 (compatible; MSIE 6.0; Windows NT 5.0; en)");
user_pref("useragentswitcher.6.description", "Opera 8.0 (Win 2000)");
user_pref("useragentswitcher.6.useragent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; en) Opera 8.0");
user_pref("useragentswitcher.7.description", "Opera 7.51 (Win XP)");
user_pref("useragentswitcher.7.useragent", "Opera/7.51 (Windows NT 5.1; U) [en]");
user_pref("useragentswitcher.8.description", "Opera 7.5 (Win XP)");
user_pref("useragentswitcher.8.useragent", "Opera/7.50 (Windows XP; U)");
user_pref("useragentswitcher.9.description", "Opera 7.5 (Win ME)");
user_pref("useragentswitcher.9.useragent", "Opera/7.50 (Windows ME; U) [en]");
user_pref("useragentswitcher.import.overwrite", false);
user_pref("useragentswitcher.menu.hide", false);
user_pref("useragentswitcher.reset.onclose", true);
user_pref("useragentswitcher.user.agents.count", 56);