{"action":"opened","issue":{"url":"https://api.github.com/repos/StewShack/PersonalWebsite/issues/115","repository_url":"https://api.github.com/repos/StewShack/PersonalWebsite","labels_url":"https://api.github.com/repos/StewShack/PersonalWebsite/issues/115/labels{/name}","comments_url":"https://api.github.com/repos/StewShack/PersonalWebsite/issues/115/comments","events_url":"https://api.github.com/repos/StewShack/PersonalWebsite/issues/115/events","html_url":"https://github.com/StewShack/PersonalWebsite/issues/115","id":245917265,"number":115,"title":"C# crawling a site with Abot","user":{"login":"StewShack","id":376435,"avatar_url":"https://avatars3.githubusercontent.com/u/376435?v=4","gravatar_id":"","url":"https://api.github.com/users/StewShack","html_url":"https://github.com/StewShack","followers_url":"https://api.github.com/users/StewShack/followers","following_url":"https://api.github.com/users/StewShack/following{/other_user}","gists_url":"https://api.github.com/users/StewShack/gists{/gist_id}","starred_url":"https://api.github.com/users/StewShack/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/StewShack/subscriptions","organizations_url":"https://api.github.com/users/StewShack/orgs","repos_url":"https://api.github.com/users/StewShack/repos","events_url":"https://api.github.com/users/StewShack/events{/privacy}","received_events_url":"https://api.github.com/users/StewShack/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-07-27T03:41:32Z","updated_at":"2017-07-27T03:41:32Z","closed_at":null,"body":"Write a page about crawling a webpage with Abot using C#.\r\n\r\nAdd Abot Nuget\r\nAdd System.Configuration\r\n\r\n```\r\nusing System;\r\nusing System.Net;\r\n \r\nusing Abot.Core;\r\nusing Abot.Crawler;\r\n \r\nnamespace StewShack.Crawler\r\n{\r\n    public class Crawler\r\n    {\r\n        public void Crawl( Uri page )\r\n        {\r\n            using ( var crawler = new PoliteWebCrawler( AbotConfigurationSectionHandler.LoadFromXml().Convert() ) )\r\n            {\r\n                crawler.PageCrawlCompletedAsync += CrawlerProcessPageCrawlCompleted;\r\n \r\n                crawler.Crawl( page );\r\n            }\r\n        }\r\n \r\n        private void CrawlerProcessPageCrawlCompleted( object sender, PageCrawlCompletedArgs e )\r\n        {\r\n            var crawledPage = e.CrawledPage;\r\n \r\n            if ( crawledPage.WebException != null\r\n                 || crawledPage.HttpWebResponse.StatusCode != HttpStatusCode.OK )\r\n            {\r\n                Console.WriteLine( \"Crawl of page failed {0}. Status Code {1}. Exception {2}\",\r\n                                   crawledPage.Uri.AbsoluteUri,\r\n                                   crawledPage.HttpWebResponse.StatusCode,\r\n                                   crawledPage.WebException );\r\n            }\r\n        }\r\n    }\r\n}\r\n \r\nusing System;\r\n \r\nusing log4net.Config;\r\n \r\nnamespace StewShack.Crawler\r\n{\r\n    public class Program\r\n    {\r\n        public static void Main( string[] args )\r\n        {\r\n            CrawlSite();\r\n        }\r\n \r\n        private static void CrawlSite()\r\n        {\r\n            XmlConfigurator.Configure();\r\n \r\n            var fullUri = new Uri( \"http://www.stewshack.com\" );\r\n \r\n            Console.WriteLine( \"Crawling: {0}\", fullUri.AbsoluteUri );\r\n \r\n            new Crawler().Crawl( fullUri );\r\n        }\r\n \r\n    }\r\n}\r\n \r\n<?xml version=\"1.0\" encoding=\"utf-8\"?>\r\n<configuration>\r\n    <configSections>\r\n        <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler, log4net\" />\r\n        <section name=\"abot\" type=\"Abot.Core.AbotConfigurationSectionHandler, Abot\" />\r\n    </configSections>\r\n    <startup>\r\n        <supportedRuntime version=\"v4.0\" sku=\".NETFramework,Version=v4.6.2\" />\r\n    </startup>\r\n    <log4net>\r\n        <appender name=\"ConsoleAppender\" type=\"log4net.Appender.ConsoleAppender\">\r\n            <layout type=\"log4net.Layout.PatternLayout\">\r\n                <conversionPattern value=\"[%date] [%thread] [%-5level] - %message%newline\" />\r\n            </layout>\r\n        </appender>\r\n        <appender name=\"AbotAppender\" type=\"log4net.Appender.RollingFileAppender\">\r\n            <file value=\"abotlog.txt\" />\r\n            <appendToFile value=\"true\" />\r\n            <rollingStyle value=\"Size\" />\r\n            <maxSizeRollBackups value=\"10\" />\r\n            <maximumFileSize value=\"10240KB\" />\r\n            <staticLogFileName value=\"true\" />\r\n            <preserveLogFileNameExtension value=\"true\" />\r\n            <layout type=\"log4net.Layout.PatternLayout\">\r\n                <conversionPattern value=\"[%date] [%-3thread] [%-5level] - %message%newline\" />\r\n            </layout>\r\n        </appender>\r\n        <logger name=\"AbotLogger\">\r\n            <level value=\"INFO\" />\r\n            <appender-ref ref=\"ConsoleAppender\" />\r\n            <appender-ref ref=\"AbotAppender\" />\r\n        </logger>\r\n    </log4net>\r\n    <abot>\r\n        <crawlBehavior crawlTimeoutSeconds=\"0\" downloadableContentTypes=\"text/html, text/plain\" httpRequestMaxAutoRedirects=\"0\" httpRequestTimeoutInSeconds=\"0\" httpServicePointConnectionLimit=\"200\" isExternalPageCrawlingEnabled=\"true\" isExternalPageLinksCrawlingEnabled=\"false\" isForcedLinkParsingEnabled=\"false\" isHttpRequestAutomaticDecompressionEnabled=\"true\" isHttpRequestAutoRedirectsEnabled=\"true\" isRespectUrlNamedAnchorOrHashbangEnabled=\"false\" isSendingCookiesEnabled=\"false\" isSslCertificateValidationEnabled=\"false\" isUriRecrawlingEnabled=\"false\" maxConcurrentThreads=\"1\" maxCrawlDepth=\"1\" maxLinksPerPage=\"0\" maxMemoryUsageCacheTimeInSeconds=\"0\" maxMemoryUsageInMb=\"0\" maxPageSizeInBytes=\"100000\" maxPagesToCrawl=\"0\" maxPagesToCrawlPerDomain=\"0\" maxRetryCount=\"1\" minAvailableMemoryRequiredInMb=\"0\" minRetryDelayInMilliseconds=\"12000\" userAgentString=\"abot v1.0 http://code.google.com/p/abot\" />\r\n        <authorization isAlwaysLogin=\"false\" loginUser=\"\" loginPassword=\"\" />\r\n        <politeness isIgnoreRobotsDotTextIfRootDisallowedEnabled=\"false\" isRespectAnchorRelNoFollowEnabled=\"true\" isRespectHttpXRobotsTagHeaderNoFollowEnabled=\"true\" isRespectMetaRobotsNoFollowEnabled=\"true\" isRespectRobotsDotTextEnabled=\"true\" maxRobotsDotTextCrawlDelayInSeconds=\"120\" robotsDotTextUserAgentString=\"abot v1.0 http://code.google.com/p/abot\" minCrawlDelayPerDomainMilliSeconds=\"1000\" />\r\n    </abot>\r\n    <runtime>\r\n        <assemblyBinding xmlns=\"urn:schemas-microsoft-com:asm.v1\">\r\n            <dependentAssembly>\r\n                <assemblyIdentity name=\"log4net\" publicKeyToken=\"669e0ddf0bb1aa2a\" culture=\"neutral\" />\r\n                <bindingRedirect oldVersion=\"0.0.0.0-2.0.7.0\" newVersion=\"2.0.7.0\" />\r\n            </dependentAssembly>\r\n        </assemblyBinding>\r\n    </runtime>\r\n</configuration>\r\n```"}}
{"actor":{"display_login":"StewShack"}}