I have converted some PDF documents which contained JPEG into ASCII text.

Unfortunately, there are many
breaks in the middle of
sentences, leaving some lines with only a few words and the
text generally difficult to read.

Is there any
tool which will reformat the
text, taking into account sentences and
which might be broken up like
this. It should
recognize English sentence structure - that they generally
end with
period, question
mark full
stop, etc

  • 1
    What types of sentence structure (e.g. period) would you like to have a break in? – Tom Jan 2 '16 at 23:17
  • I would like the program to understand the grammar of the English language. – Mawg Jan 3 '16 at 10:28

You can do this in most editing software (word processors/text editors) that lets you search/replace paragraph marks or line breaks. Here's is an example strategy described in Word syntax where ^p is a paragraph break, assuming no distinction between new-line characters and page breaks (which is the case for text files), and in which I will use _ for space characters.

First insert extra paragraph marks after sentence punctuation:

  • Replace . with .^p
  • Replace ? with ?^p
  • Replace ! with !^p

Remove leading and trailing spaces, keep repeating until no more found:

  • Replace _^p with ^p
  • Replace ^p_ with ^p

Reduce the number of consecutive paragraph breaks to two at the most. Keep repeating until no more found:

  • Replace ^p^p^p with ^p^p

Additional steps if you want to stitch sentences together into one parapraph:

  • Replace ^p^p with $$##$$ (or any string that is not in your document)
  • Replace ^p with _
  • Replace $$##$$ with ^p

Additional notes:

  • Put all this in a keyboard macro if you need to do it in multiple documents.
  • You may wish to reduce to one paragraph break only. Then you need a variation of this strategy.
  • You can use regular expressions as well, but I'm not fluent with them.
  • For that additional step your software must be able to handle very long lines, because you will have an intermediate result which is basically just one line.
  • Do all this on copies of your files so that you have a backup.
  • Much as I would prefer a program to do it, I had considered this, so +1. I would need to use a text editor which allows macros, so that I can store that sequence and repeat it for many files. Fortunately, NotePad++ will let me do so. – Mawg Jan 11 '16 at 20:06

Your Answer

