Postgres server performance falls off a cliff after a certain number of records
===

I am using a cursor to retrieve records from a large postgres table. (400 million records, using child tables to partition the data.) My cursor is defined as "select * from parent_table order by indexed_column". 

Using both JDBC and psql, performance is consistent for the first few hundred thousand retrievals. After that it falls off a cliff and never recovers. On the server CPU, memory, and disk activity are fairly even; i.e. nothing system based stands out as an obvious culprit. I was initially suspecting this might be a network issue, but I've reproduced this from different networks.

Here's psql:

    db@dbdev> fetch 100000 from all_persons;
    Time: 13995.910 ms
    db@dbdev> fetch 100000 from all_persons;
    Time: 13852.955 ms
    db@dbdev> fetch 100000 from all_persons;
    Time: 14037.631 ms
    db@dbdev> fetch 100000 from all_persons;
    Time: 13818.516 ms
    db@dbdev> fetch 100000 from all_persons;
    Time: 13952.260 ms
    db@dbdev> fetch 100000 from all_persons;
    Time: 14257.836 ms
    db@dbdev> fetch 100000 from all_persons;
    Time: 14115.941 ms
    db@dbdev> fetch 100000 from all_persons;
    Time: 14375.485 ms
    db@dbdev> fetch 100000 from all_persons;
    Time: 14898.741 ms
    db@dbdev> fetch 100000 from all_persons;
    Time: 14086.004 ms
    db@dbdev> fetch 100000 from all_persons;
    Time: 59841.556 ms
    db@dbdev> fetch 100000 from all_persons;
    Time: 198176.211 ms
    db@dbdev> fetch 100000 from all_persons;
    Time: 162593.582 ms

And here's JDBC (retrieving 10000 at a time; the number at the left is the count of the filtered set of records which are inserted back):

    ...
    536040 retrieve in 405; filtering in 28; insert in 1734
    544739 retrieve in 413; filtering in 27; insert in 1713
    553574 retrieve in 382; filtering in 27; insert in 1761
    563167 retrieve in 348; filtering in 28; insert in 2019
    572723 retrieve in 363; filtering in 27; insert in 2048
    581736 retrieve in 363; filtering in 28; insert in 1784
    591131 retrieve in 480; filtering in 28; insert in 1869
    600260 retrieve in 377; filtering in 27; insert in 1831
    608234 retrieve in 24074; filtering in 27; insert in 1566
    616212 retrieve in 23711; filtering in 27; insert in 1649
    624449 retrieve in 25913; filtering in 27; insert in 1587
    632528 retrieve in 29981; filtering in 27; insert in 1527
    641334 retrieve in 23231; filtering in 27; insert in 1728
    650427 retrieve in 27883; filtering in 27; insert in 1996
    659516 retrieve in 34422; filtering in 27; insert in 1774

While the psql performance appears to be getting worse, the JDBC performance stays roughly consistent at least through a million records (bouncing between approx 34k and 17k milliseconds).

What explains the sudden drop in performance?
