title: Undeclared general entity error while performing XML full import on Solr using DIH
body: <p>I am trying to bulk import xml files from a folder to solr.
  My DIH config looks like this.</p>
  
  <pre class="lang-html prettyprint-override"><code>&lt;dataConfig&gt;
    &lt;dataSource type="FileDataSource"/&gt;
    &lt;document&gt;
      &lt;!-- this outer processor generates a list of files satisfying the conditions
           specified in the attributes --&gt;
      &lt;entity name="xmlImport" processor="FileListEntityProcessor"
              fileName=".*xml"
              recursive="true"
              rootEntity="false"
              dataSource="null"
              baseDir="/home/rsp/shellscript/output"
            &gt;
  
        &lt;!-- this processor extracts content using Xpath from each file found --&gt;
  
        &lt;entity name="nested" processor="XPathEntityProcessor" transformer="DateFormatTransformer,TemplateTransformer"
                forEach="/root" url="${xmlImport.fileAbsolutePath}"&gt;
  
  
          &lt;field column="id" xpath="/root/sub1/sub2/id"/&gt;
  
        &lt;/entity&gt;
      &lt;/entity&gt;
    &lt;/document&gt;
  &lt;/dataConfig&gt;
  </code></pre>
  
  <p>when i am trying to run the full import, the job fails leaving the below log.</p>
  
  <pre class="lang-js prettyprint-override"><code>Full Import failed:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException: Parsing failed for xml, url:/home/rsp/shellscript/output/file1.xml rows processed:0 Processing Document # 1
      at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:271)
      at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:417)
      at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:481)
      at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:462)
  Caused by: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException: Parsing failed for xml, url:/home/rsp/shellscript/output/file1.xml rows processed:0 Processing Document # 1
      at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:417)
      at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:330)
      at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:233)
      ... 3 more
  Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException: Parsing failed for xml, url:/home/rsp/shellscript/output/file1.xml rows processed:0 Processing Document # 1
      at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:70)
      at org.apache.solr.handler.dataimport.XPathEntityProcessor.initQuery(XPathEntityProcessor.java:330)
      at org.apache.solr.handler.dataimport.XPathEntityProcessor.fetchNextRow(XPathEntityProcessor.java:225)
      at org.apache.solr.handler.dataimport.XPathEntityProcessor.nextRow(XPathEntityProcessor.java:205)
      at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:244)
      at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:476)
      at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:515)
      at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415)
      ... 5 more
  Caused by: java.lang.RuntimeException: com.ctc.wstx.exc.WstxParsingException: Undeclared general entity "ldquo"
   at [row,col {unknown-source}]: [147,57]
      at org.apache.solr.handler.dataimport.XPathRecordReader.streamRecords(XPathRecordReader.java:188)
      at org.apache.solr.handler.dataimport.XPathEntityProcessor.initQuery(XPathEntityProcessor.java:319)
      ... 11 more
  Caused by: com.ctc.wstx.exc.WstxParsingException: Undeclared general entity "ldquo"
   at [row,col {unknown-source}]: [147,57]
      at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:614)
      at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:487)
      at com.ctc.wstx.sr.BasicStreamReader.handleUndeclaredEntity(BasicStreamReader.java:5470)
      at com.ctc.wstx.sr.StreamScanner.expandUnresolvedEntity(StreamScanner.java:1742)
      at com.ctc.wstx.sr.StreamScanner.expandEntity(StreamScanner.java:1626)
      at com.ctc.wstx.sr.StreamScanner.fullyResolveEntity(StreamScanner.java:1564)
      at com.ctc.wstx.sr.BasicStreamReader.skipTokenText(BasicStreamReader.java:3604)
      at com.ctc.wstx.sr.BasicStreamReader.skipToken(BasicStreamReader.java:3369)
      at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2629)
      at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1073)
      at org.apache.solr.handler.dataimport.XPathRecordReader$Node.handleStartElement(XPathRecordReader.java:377)
      at org.apache.solr.handler.dataimport.XPathRecordReader$Node.parse(XPathRecordReader.java:311)
      at org.apache.solr.handler.dataimport.XPathRecordReader$Node.handleStartElement(XPathRecordReader.java:347)
      at org.apache.solr.handler.dataimport.XPathRecordReader$Node.parse(XPathRecordReader.java:311)
      at org.apache.solr.handler.dataimport.XPathRecordReader$Node.handleStartElement(XPathRecordReader.java:347)
      at org.apache.solr.handler.dataimport.XPathRecordReader$Node.parse(XPathRecordReader.java:311)
      at org.apache.solr.handler.dataimport.XPathRecordReader$Node.handleStartElement(XPathRecordReader.java:347)
      at org.apache.solr.handler.dataimport.XPathRecordReader$Node.parse(XPathRecordReader.java:311)
      at org.apache.solr.handler.dataimport.XPathRecordReader$Node.access$200(XPathRecordReader.java:203)
      at org.apache.solr.handler.dataimport.XPathRecordReader.streamRecords(XPathRecordReader.java:185)
      ... 12 more
  </code></pre>
  
  <h1>"Undeclared general entity 'ldquo' in the xml".</h1>
  
  <p>I am having xhtml entities like &amp;ldquo,&amp;reg inside the xml which are not required for indexing.
  How can i ignore this and retrieve the field values from the xml?
  I'm using solr 5.4.1 instance and java version "1.7.0_45".</p>