title: Pandas skipping malformed line in csv
body: <p>I am trying to read a csv file with pandas. the file is very long and malformed in the middle like so</p>
  
  <pre><code>Date,Received Date,Tkr,Theta,Wid,Per
  2007-08-03,2017/02/13 05:30:G,F,B,A,1
  2007-08-06,2017/02/13 05:30:G,F,A,B,1
  2007-08-07,2017/02/13 05:30:G,F,A,B,1
  2007-08-,nan,,,,
  2000-05-30 00:00:00,2017/02/14 05:30:F,D,10,1,1
  2000-05-31 00:00:00,2017/02/14 05:30:F,D,10,1,1
  </code></pre>
  
  <p>My line which is failing is this:</p>
  
  <pre><code>full_frame = pd.read_csv(path, parse_dates=["Date"],error_bad_lines=False).set_index("Date").sort_index()[:date]
  </code></pre>
  
  <p>with the error</p>
  
  <pre><code>TypeError: unorderable types: str() &gt; datetime.datetime()
     File "/A/B/C.py", line 236, in load_ex
      full_frame = pd.read_csv(path, parse_dates=["Date"],error_bad_lines=False).set_index("Date").sort_index()[:date]
  </code></pre>
  
  <p><code>date</code> is just a variable that holds a given input date.</p>
  
  <p>This is happening because of the broken line in the middle. I have tried to do</p>
  
  <p><code>error_bad_line=False</code> but that wont prevent my script from failing.</p>
  
  <p>When i take out the bad line from my csv and run it, it works fine. This csv will be used as an input and I cant modify it at source so I was wondering if there is a way to skip a line based on length of the line in the csv in pandas or something else I can do to make it work without duplicating/modifyng the file </p>
  
  <p>UPDATE</p>
  
  <p>The bad line is stored in my data frame if i simply do a </p>
  
  <pre><code>read_csv 
  </code></pre>
  
  <p>as <code>2007-08-           NaN    NaN   NaN      NaN     NaN</code></p>
  
  <p>UPDATE 2:</p>
  
  <p>if i try to just do </p>
  
  <pre><code>full_frame = pd.read_csv(path, parse_dates=["Date"],error_bad_lines=False)
  full_frame = full_frame.dropna(how="any")
  # this drops the NaN row for sure
  full_frame = full_frame.set_index("Date").sort_index()[:date]
  </code></pre>
  
  <p>still gives same error :(</p>