blenderNetworkpatreonblender-logosdheartuserthtimesbookphotopencilmap-markerchevron-rightbancommentcommentstwitterfacebookrsschainnavicongoogle-plusenvelopelinkedinmail-replycalendar-otumblrredditstumbleuponshare-altbloggerflattrflag

Making a web crawler

Making a web crawler

Postby themitchel0 » February 12th, 2011, 9:07 am

hello again :D does any one know how to make a web crawler, a simple one, that would just gather links and store them in a file? preferably in php, so it can be hosted on a server? thanks!
themitchel0
Pro
Pro
 
Posts: 365
Joined: June 15th, 2010, 8:31 am

Re: Making a web crawler

Postby IrascibleOne » February 13th, 2011, 1:54 pm

Hmmm, I dunno, I never looked up much of anything about making web crawlers. I know it can be done it php, as I am pretty sure I have seen it mentioned somewhere before but other than that, I don't think I can help much.

Off the top of my head it sounds like you need to go to a page and read the HTML in the page - assuming the page has HTML. I do know that you can read HTML with php, in particular you could potentially find all links (that are not javascript links) using PHP DOM. Beyond that, this tutorial might help some.
Image
User avatar
IrascibleOne
Site Admin
Site Admin
 
Posts: 1540
Joined: July 31st, 2008, 7:14 pm

Re: Making a web crawler

Postby themitchel0 » February 16th, 2011, 3:54 pm

ok thanks!
themitchel0
Pro
Pro
 
Posts: 365
Joined: June 15th, 2010, 8:31 am
Return to Programming

Who is online

Users browsing this forum: No registered users and 1 guest