Hmmm, I dunno, I never looked up much of anything about making web crawlers. I know it can be done it php, as I am pretty sure I have seen it mentioned somewhere before but other than that, I don't think I can help much.
. Beyond that, this tutorial
might help some.