title: Java How to read a TEXT log file that contains separate XML and extract the XML Blocks
body: <p>Using Java</p>
  
  <p>I have an application that writes all the XML messages it receives into Log files.  Each XML message is appended to the log file by log4j.</p>
  
  <p>This should work for any type of text file that has a unique start and end of a block of text.  This is not specific.</p>
  
  <p>There can be thousands of XML messages in each log file and new log files are created each day.</p>
  
  <p>I have no need to parse the XML in the log files into it's elements other than to pull each XML message (block of XML) from the log file into a memory variable in Java.  (The entire block of XML from the first tag to the last needs to be in a memory variable).</p>
  
  <p>I will be sending this XML to a Web Service to be processed similar to the way it is being sent over from our Middleware today.</p>
  
  <p>I already have that part working to send the XML and am able to process files as long as the XML is on one row in the log file but the XML writer in the application writs the XML in some cases in an indented new line format hence the need to be able to pluck the blocks of XML messages from the log file.</p>
  
  <p>So -I need to read the XML in the log file from the first tag:</p>
  
  <pre><code>&lt;?xml version='1.0' encoding='UTF-8'?&gt;
  </code></pre>
  
  <p>Until the last tag of the last tag: </p>
  
  <pre><code>&lt;/MyXML&gt;
  </code></pre>
  
  <p>No matter if there are carriage returns or blank lines in the file and it has to skip invalid rows.</p>
  
  <p>Each XML message can be small to very large - easily over 20k.</p>
  
  <p>The XML log file will look something like the following and may have blank rows between each XML message or other text as shown below:</p>
  
  <pre><code>&lt;?xml version='1.0' encoding='UTF-8'?&gt;
  &lt;MyXML&gt;
  &lt;Envelope documentType="SetProfile" trader="BEA" dtdRev="2.0" xid="03-JUL-17 14:38:49" traderLogin="middleware" traderPassword="abc123"/&gt;
  &lt;Payload&gt;&lt;SetProfile allowInvalidProfile="F"&gt;
  &lt;Partner publisherID="52725" act="Update"&gt;
  &lt;Contact languageCode="EN" firstName="Luis" lastName="Dini" email="Dini@email.com" act="Update" publisherID="ldini" securityRoleCode="6"/&gt;
  &lt;/Partner&gt;&lt;/SetProfile&gt;
  &lt;/Payload&gt;
  &lt;/MyXML&gt;
  
  &lt;?xml version='1.0' encoding='UTF-8'?&gt;&lt;MyXML&gt;&lt;Envelope documentType="SetProfile" trader="BEA" dtdRev="2.0" xid="03-JUL-17 14:38:49" traderLogin="middleware" traderPassword="abc123"/&gt;&lt;Payload&gt;&lt;SetProfile allowInvalidProfile="F"&gt;&lt;Partner publisherID="9857684" act="Update"&gt;&lt;Contact languageCode="EN" firstName="Bill" lastName="Jones" email="Jones@email.com" act="Update" publisherID="BJones" securityRoleCode="3"/&gt;&lt;/Partner&gt;&lt;/SetProfile&gt;&lt;/Payload&gt;&lt;/MyXML&gt;
  
  ======================
  #]
  &lt;?xml version='1.0' encoding='UTF-8'?&gt;&lt;MyXML&gt;&lt;Envelope documentType="SetProfile" trader="BEA" dtdRev="2.0" xid="03-JUL-17 14:38:49" traderLogin="middleware" traderPassword="abc123"/&gt;
  &lt;Payload&gt;&lt;SetProfile allowInvalidProfile="F"&gt;&lt;Partner publisherID="7465737" act="Update"&gt;
  &lt;Contact languageCode="EN" firstName="John" lastName="Smith" email="Smith@email.com" act="Update" publisherID="JSmith" securityRoleCode="3"/&gt;&lt;/Partner&gt;&lt;/SetProfile&gt;&lt;/Payload&gt;&lt;/MyXML&gt;
  </code></pre>
  
  <p>In short this will be a tool that will read a log file of XML messages and extract each individual XML message to be forwarded to a Web Service similar to the way the middleware is creating and sending each XML message today.</p>
  
  <p>This will be used for volume testing and other development needs.</p>
  
  <p>Any suggestions are appreciated.</p>