title: How to remove weird encoding from txt file
body: <p>I am trying to process text files like this one:</p>
  
  <p><a href="http://www.sec.gov/Archives/edgar/data/789019/000119312514289961/0001193125-14-289961.txt" rel="nofollow">http://www.sec.gov/Archives/edgar/data/789019/000119312514289961/0001193125-14-289961.txt</a></p>
  
  <p>If you see around the middle of the file there is something like the following:</p>
  
  <pre><code>&lt;/TEXT&gt;
  &lt;/DOCUMENT&gt;
  &lt;DOCUMENT&gt;
  &lt;TYPE&gt;EXCEL
  &lt;SEQUENCE&gt;21
  &lt;FILENAME&gt;Financial_Report.xlsx
  &lt;DESCRIPTION&gt;IDEA: XBRL DOCUMENT
  &lt;TEXT&gt;
  begin 644 Financial_Report.xlsx
  M4$L#!!0`!@`(````(0!):[_C#0,``+!)```3``@"6T-O;G1E;G1?5'EP97-=
  M+GAM;""B!`(HH``"````````````````````````````````````````````
  M````````````````````````````````````````````````````````````
  M````````````````````````````````````````````````````````````
  M````````````````````````````````````````````````````````````
  M````````````````````````````````````````````````````````````
  M````````````````````````````````````````````````````````````
  M````````````````````````````````````````````````````````````
  M````````````````````````````````````````````````````````````
  M````````````````````````````````````````````````````````````
  M````````````````````````````````````````````````````````````
  M````````````````````````````````````````````````````````````
  M``````````````````````````````````````#,W,M.VT`4QO%]I;Z#Y6V5
  M&gt;([OK@@L&gt;EFV2*4/,+4GQ,(W&gt;08*;]^)N0BA%(2*U/^&amp;B,2&gt;\\6+G[+YSM')
  M==\%5V:V[3AL0EFK,#!#/3;M&lt;+X)?YY]795A8)T&gt;&amp;MV-@]F$-\:&amp;)\?OWQV=
  MW4S&amp;!O[NP6["G7/3QRBR]&lt;[TVJ['R0S^D^TX]]KY?^?S:-+UA3XW4:Q4'M7C
  MX,S@5FY_1GA\]-EL]67G@B_7_NW;)+/I;!A\NKUP/VL3ZFGJVEH[GS2Z&amp;IHG
  M4U9W$];^SN4:NVLG^\''"*.#$_:?_'W`W7W?_:.9V\8$IWIVWW3O8T377?1[
  MG"]^C&gt;/%^OE##J0&lt;M]NV-LU87_;^":SM-!O=V)TQKN_6R^NZU^UPG_N9^&lt;O%
  M-EI&gt;Y(V#[+_?&lt;O`K&lt;\20'`DD1PK)D4%RY)`&lt;!21'"&lt;E107*(H@2AB"H44H5B
  MJE!0%8JJ0F%5**X*!5:AR!I39(TILL8466.*K#%%UI@B:TR1-:;(&amp;E-DC2FR
  M)A19$XJL"476A")K0I$UH&lt;B:4&amp;1-*+(F%%D3BJPI1=:4(FM*D36ER)I29$TI
  MLJ8465.*K"E%UI0B:T:1-:/(FE%DS2BR9A19,XJL&amp;476C")K1I$UH\B:4V3-
  M*;+F%%ESBJPY1=:&lt;(FM.D36GR)I39,TILA8460N*K`5%UH(B:T&amp;1M:#(6E!D
  M+2BR%A19"XJL)476DB)K29&amp;UI,A:4F0M*;*6%%E+BJPE1=:2(FM%D;6BR%I1
  M9*THLE8462N*K!5%UHHB:T61M:+(*HI"JRB*K:(HN(JBZ"J*PJLHBJ^B*,"*
  MH@@KBD*L*(RQH#H6QEA.(8O3R.)4LCB=+$XIB]/*XM2R,+TLP12S!-/,$DPU
  M2S#=+,&amp;4LP33SA),/4LP_2S!%+0$T]"2_U;1&lt;GX?CHF6O__^`W8YYH6%+-;=
  M=,:^\1*%VT-?FKS3LVE^N-EO#GKS`(_/?BZ'WZMS.H^3]1N&amp;9O/ZIW"_0FA_
  M]VKR!YG9M&gt;9AB="A93P/$_UVHM&lt;/?+(-R.SW'S6F.3`[6O8M'?\!``#__P,`
  M4$L#!!0`!@`(````(0"U53`C]0```$P"```+``@"7W)E;',O+G)E;',@H@0"
  </code></pre>
  
  <p>This seems like an excel file ? Or an XBRL document ? What is that ? How do I get rid of it (or "process" it somehow??) This goes on for thousands of lines so I guess it is some encoding of some link of some attached file?? Any idea how to deal with it ?</p>
  
  <p>I am trying to use BeautifulSoup in Python:</p>
  
  <pre><code>from bs4 import BeautifulSoup
  
  with open("textWithHtml.txt") as markup:
      soup = BeautifulSoup(markup.read())
  
  with open("processedText.txt", "w") as f: 
      f.write(soup.get_text().encode('utf-8'))
  </code></pre>
  
  <p>but not everything is removed, and also I noticed that in some cases not even all html tags are removed.. sometimes running the code twice removes more than what were removed the first time the BeautifulSoup code was run..</p>