C# URL Crawler not getting enough links?
===

I have the following code, however, when I launch it I only ever seam to get a few URLS returned.

    while (stopFlag != true)
                        {
                            WebRequest request = WebRequest.Create(urlList[i]);
                            using (WebResponse response = request.GetResponse())
                            {
                                using (StreamReader reader = new StreamReader
                                   (response.GetResponseStream(), Encoding.UTF8))
                                {
                                    string sitecontent = reader.ReadToEnd();
                                    //add links to the list
                                    // process the content
                                    //clear the text box ready for the HTML code
                                    //Regex urlRx = new Regex(@"((https?|ftp|file)\://|www.)[A-Za-z0-9\.\-]+(/[A-Za-z0-9\?\&\=;\+!'\(\)\*\-\._~%]*)*", RegexOptions.IgnoreCase);
                                    Regex urlRx = new Regex(@"(?<url>(http:[/][/]|www.)([a-z]|[A-Z]|[0-9]|[/.]|[~])*)", RegexOptions.IgnoreCase);
                                    
                                    MatchCollection matches = urlRx.Matches(sitecontent);
                                    foreach (Match match in matches)
                                    {
                                        string cleanMatch = cleanUP(match.Value);
                                        urlList.Add(cleanMatch);

                                        updateResults(theResults, "\"" + cleanMatch + "\",\n");

                                    }
                                }
                            }
                        }

I think the error is within the regex.

What I am trying to achieve is pull a webpage, then grab all the links from that page - add these to a list, then for each list item fetch the next page and repeat the process.