{"action":"closed","issue":{"url":"https://api.github.com/repos/Norconex/collector-http/issues/370","repository_url":"https://api.github.com/repos/Norconex/collector-http","labels_url":"https://api.github.com/repos/Norconex/collector-http/issues/370/labels{/name}","comments_url":"https://api.github.com/repos/Norconex/collector-http/issues/370/comments","events_url":"https://api.github.com/repos/Norconex/collector-http/issues/370/events","html_url":"https://github.com/Norconex/collector-http/issues/370","id":248761844,"number":370,"title":"Eliminate header and footer data  form crawled data.","user":{"login":"Navaminavu","id":29800957,"avatar_url":"https://avatars2.githubusercontent.com/u/29800957?v=4","gravatar_id":"","url":"https://api.github.com/users/Navaminavu","html_url":"https://github.com/Navaminavu","followers_url":"https://api.github.com/users/Navaminavu/followers","following_url":"https://api.github.com/users/Navaminavu/following{/other_user}","gists_url":"https://api.github.com/users/Navaminavu/gists{/gist_id}","starred_url":"https://api.github.com/users/Navaminavu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Navaminavu/subscriptions","organizations_url":"https://api.github.com/users/Navaminavu/orgs","repos_url":"https://api.github.com/users/Navaminavu/repos","events_url":"https://api.github.com/users/Navaminavu/events{/privacy}","received_events_url":"https://api.github.com/users/Navaminavu/received_events","type":"User","site_admin":false},"labels":[{"id":29162741,"url":"https://api.github.com/repos/Norconex/collector-http/labels/question","name":"question","color":"fbca04","default":true}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2017-08-08T15:50:05Z","updated_at":"2017-11-23T18:29:40Z","closed_at":"2017-11-23T18:29:39Z","author_association":"NONE","body":"Hi Pascal,\r\nCan you help me to avoid the header and footer data from a page  being crawled\r\n\r\nPlease find below the\r\n[htmlfile _l2tm.txt](https://github.com/Norconex/collector-http/files/1208703/htmlfile._l2tm.txt)\r\n config file\r\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\n<httpcollector id=\"HTTPCollectorlmr\">\r\n\r\n\t#set($http = \"com.norconex.collector.http\")\r\n\t#set($core = \"com.norconex.collector.core\")\r\n\t#set($urlNormalizer   = \"${http}.url.impl.GenericURLNormalizer\")\r\n\t#set($filterExtension = \"${core}.filter.impl.ExtensionReferenceFilter\")\r\n\t#set($filterRegexRef  = \"${core}.filter.impl.RegexReferenceFilter\")\r\n\t#set($http\r\n\t= \"com.norconex.collector.http\")\r\n\t#set($committer =\r\n\t\"com.norconex.committer\")\r\n\t#set($urlNormalizer   = \"${http}.url.impl.GenericURLNormalizer\")\r\n\t#set($filterExtension = \"${core}.filter.impl.ExtensionReferenceFilter\")\r\n\t#set($sitemapFactory    = \"${http}.sitemap.impl.StandardSitemapResolverFactory\")\r\n\r\n    <crawlerDefaults>\r\n        <numThreads>2</numThreads>\r\n        <orphansStrategy>DELETE</orphansStrategy>\r\n        <delay default=\"500\" />\r\n        <robotsTxt ignore=\"false\"\r\n\t\t\t\tclass=\"com.norconex.collector.http.robot.impl.StandardRobotsTxtProvider\" />\r\n         <referenceFilters>\r\n            <filter class=\"$filterExtension\" onMatch=\"exclude\">jpg,gif,png,ico,css,js\r\n\t\t\t</filter>\r\n        </referenceFilters>\r\n    </crawlerDefaults>\r\n    <crawlers>\r\n        <crawler id=\"lmr\">\r\n         <sitemapResolverFactory ignore=\"true\" lenient=\"true\" class=\"com.norconex.collector.http.sitemap.impl.StandardSitemapResolverFactory\"/>\r\n\t\t\t<startURLs stayOnDomain=\"true\" stayOnPort=\"true\" stayOnProtocol=\"true\">\r\n\t\t\t<url>https://lmr-doxygen.rnd.ki.sw.ericsson.se/l2tm/</url>\t\t\r\n\t\t\t</startURLs>\t\t\t\r\n            <canonicalLinkDetector class=\"com.norconex.collector.http.url.impl.GenericCanonicalLinkDetector\" ignore=\"true\"></canonicalLinkDetector>\r\n            <!-- Where the crawler default directory to generate files is. -->\r\n            <workDir>./output/lmr</workDir>\r\n            <maxDepth>2</maxDepth>\r\n            <maxDocuments>-1</maxDocuments>\r\n            <!-- Decide where to store generated files. -->\r\n            <progressDir>./output/lmr/progress</progressDir>\r\n            <logsDir>./output/lmr/logs</logsDir>\r\n\t\t\t <!--<linkExtractors>\t\r\n\t\t\t\t<extractor class=\"com.norconex.collector.http.url.impl.GenericLinkExtractor\">\t\t\r\n\t\t\t\t\t<tags>\r\n\t\t\t\t\t\t<tag name=\"a\" attribute=\"href\" />\r\n\t\t\t\t\t\t<tag name=\"script\" attribute=\"src\" />\r\n\t\t\t\t\t</tags>\r\n\t\t\t\t</extractor>\t\t\t\t\r\n\t\t\t</linkExtractors>-->\r\n\t\t\t<documentFetcher class=\"com.norconex.collector.http.fetch.impl.PhantomJSDocumentFetcher\">\r\n\t\t\t\t<exePath>C:\\Users\\enavami\\Documents\\norconex-collector-http-2.6.2\\phantomjs-2.1.1-windows\\bin\\phantomjs.exe</exePath>\r\n\t\t\t\t<renderWaitTime>5000</renderWaitTime>\r\n\t\t\t\t<referencePattern>^.*\\.html$</referencePattern>\r\n\t\t\t</documentFetcher>\r\n\t\t\t\r\n\t\t\t\t<preParseHandlers>\r\n\t\t\t\t\t\r\n\t\t\t\t\t<transformer\t\t\t\t\r\n\t\t\t\t\t\tclass=\"com.norconex.importer.handler.transformer.impl.StripBetweenTransformer\"\r\n\t\t\t\t\t\tinclusive=\"true\" caseSensitive=\"false\" maxReadSize=\"1000000\">\r\n\t\t\t\t\t\t<stripBetween>\r\n\t\t\t\t\t\t\t<start>&lt;div class=\"title\"&gt;</start>\r\n\t\t\t\t\t\t\t<end>&lt;/div&gt;</end>\r\n\t\t\t\t\t\t</stripBetween>\r\n\t\t\t\t\t\t<stripBetween>\r\n\t\t\t\t\t\t\t<start>&lt;div class=\"contents\"&gt;</start>\r\n\t\t\t\t\t\t\t<end>&lt;/div&gt;</end>\r\n\t\t\t\t\t\t</stripBetween>\r\n\t\t\t\t\t\t<!--<stripBetween>\r\n\t\t\t\t\t\t\t<start>&lt;div id=\"main-nav\"&gt;</start>\r\n\t\t\t\t\t\t\t<end>&lt;div class=\"ui-resizable-handle ui-resizable-e\"&gt;</end>\r\n\t\t\t\t\t\t</stripBetween>-->\r\n\t\t\t\t\t</transformer>\r\n\t\t\t\t</preParseHandlers>\r\n\t\t\t\t<postParsehandlers>\r\n\t\t\t\t<transformer\r\n\t\t\t\t\t\tclass=\"com.norconex.importer.handler.transformer.impl.ReduceConsecutivesTransformer\">\r\n\t\t\t\t\t\t<reduce>\\s</reduce>\r\n\t\t\t\t\t\t<reduce>\\n</reduce>\r\n\t\t\t\t\t\t<reduce>\\s\\n</reduce>\r\n\t\t\t\t\t</transformer>\r\n\t\t\t\t</postParsehandlers>\r\n\t\t\t</importer>\r\n           \r\n            <!--   <committer class=\"com.norconex.committer.i3.I3Committer\">\r\n                <i3Params>\r\n                    <param name=\"indexURL\">http://esesslx0790.ss.sw.ericsson.se:8080/rest/Subdomain/documents.json</param>\r\n                    <param name=\"Content-type\">application/json;charset=utf-8</param>\r\n                    <param name=\"custom_id\">false</param>\r\n                    <param name=\"id_prefix\">dummy</param>\r\n                    <param name=\"id_field_name\">dummy</param>\r\n                </i3Params>\r\n                <commitBatchSize>10</commitBatchSize>\r\n                <queueDir>./queue/lmr/</queueDir>\r\n                <queueSize>10</queueSize>\r\n                <maxRetries>2</maxRetries>\r\n                <maxRetryWait>500</maxRetryWait>\r\n            </committer> \t\t-->\r\n\t\t\t<committer class=\"com.norconex.committer.core.impl.FileSystemCommitter\">\r\n\t\t\t<directory>./output/lmr/crawledFiles</directory>\r\n\t\t\t</committer>\r\n\t\t\t\r\n        </crawler>\r\n    </crawlers>\r\n</httpcollector>\r\n attaching the html file of the page and screen shot where in the area to be removed from crawl is highlighted\r\n\r\n![screenshot](https://user-images.githubusercontent.com/29800957/29081049-1e61213e-7c7f-11e7-98d9-63d8069899be.png)\r\n[htmlfile _l2tm.txt](https://github.com/Norconex/collector-http/files/1208710/htmlfile._l2tm.txt)\r\n\r\n"}}
{"actor":{"display_login":"essiembre"}}