title: pg_restore: [archiver] unsupported version (1.14) in file header
body: <p>I have a live server and development box, call them live and dev respectively both running postgresql. I can see both and manage both with pgadmin4 without trouble, and both are fully functional the one behing a live website and the other when I run by website in debug mode on the my dev box. Pretty ordinary setup.</p>
  
  <p>For years I have been running the same bash script I wrote that dumps the live database then restores it on the dev box so I have the latest live snapshot to work with.</p>
  
  <p>Today this fails me with the titled message:</p>
  
  <pre><code>pg_restore: [archiver] unsupported version (1.14) in file header
  </code></pre>
  
  <p>I have tried to diagnose this, and searched extensively on-line but am bedeviled and have failed so here I am cap in hand for expertise.</p>
  
  <p>To help I will share the following:</p>
  
  <pre><code>$ pg_dump --version
  pg_dump (PostgreSQL) 10.11 (Ubuntu 10.11-1.pgdg18.04+1)
  $ pg_restore --version
  pg_restore (PostgreSQL) 10.11 (Ubuntu 10.11-1.pgdg18.04+1)
  $ pg_dump --host=live.lan --port=5432 --dbname=mydb --username=myuser --format=custom &gt; test.backup
  $ ls -l test.backup 
  -rw-r--r-- 1 bernd bernd 2398358 Dec 23 23:40 test.backup
  $ file test.backup 
  test.backup: PostgreSQL custom database dump - v1.14-0
  $ pg_restore --dbname=mydb test.backup 
  pg_restore: [archiver] unsupported version (1.14) in file header
  </code></pre>
  
  <p>Given pg_dump and pg_restore are identical versions and:</p>
  
  <pre><code>$ which pg_dump
  /usr/bin/pg_dump
  $ which pg_restore
  /usr/bin/pg_restore
  $ ls -l /usr/bin/pg_dump /usr/bin/pg_restore
  lrwxrwxrwx 1 root root 37 Nov 14 23:23 /usr/bin/pg_dump -&gt; ../share/postgresql-common/pg_wrapper
  lrwxrwxrwx 1 root root 37 Nov 14 23:23 /usr/bin/pg_restore -&gt; ../share/postgresql-common/pg_wrapper
  </code></pre>
  
  <p>I can see they are not just identical versions but are run by the same wrapper script (which happens to be a perl script - now that's a language you don't see much anymore and I used to code in extensively)</p>
  
  <p>So I'm left totally perplexed. Thinking there may be a version issue with the live machine:</p>
  
  <pre><code>$ ssh live.lan
  Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-72-generic x86_64)
  $ which pg_dump
  /usr/bin/pg_dump
  $ which pg_restore
  /usr/bin/pg_restore
  $ pg_dump --version
  pg_dump (PostgreSQL) 10.10 (Ubuntu 10.10-0ubuntu0.18.04.1)
  $ pg_restore --version
  pg_restore (PostgreSQL) 10.10 (Ubuntu 10.10-0ubuntu0.18.04.1)
  </code></pre>
  
  <p>I can see the live box does have slightly older version of pg_dump (which would only matter if pg_dump on my dev box somehow used a RPC to the live box to run its pg_dump).</p>
  
  <p>Now there is maybe a small clue in the fact that my dev box has seen a few postgresql upgrades come through and so for example:</p>
  
  <pre><code>$ pg_lsclusters
  Ver Cluster Port Status Owner    Data directory              Log file
  10  main    5432 online postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log
  11  main    5433 online postgres /var/lib/postgresql/11/main /var/log/postgresql/postgresql-11-main.log
  12  main    5434 online postgres /var/lib/postgresql/12/main /var/log/postgresql/postgresql-12-main.log
  </code></pre>
  
  <p>The 11 and 12 clusters remain unused as evidenced by empty log files. I'm using 10. But I do notice that:</p>
  
  <pre><code>$ psql --version
  psql (PostgreSQL) 12.1 (Ubuntu 12.1-1.pgdg18.04+1)
  $ ssh live.lan
  Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-72-generic x86_64)
  $ psql --version
  psql (PostgreSQL) 10.10 (Ubuntu 10.10-0ubuntu0.18.04.1)
  </code></pre>
  
  <p>which is mildly fishy but again not obviously a cause or related:</p>
  
  <ol>
  <li>I am using pg_dump not psql</li>
  <li>I am using only the dev boxes pg tools not the live boxes (they should be irrelevant, the whole data transfer theoretically over port 5432 on the live box which delivers a databse dump to pg_dump on my dev box.</li>
  </ol>
  
  <p>Here are the clusters on the love box and it's over port 5432 that on live.lan I'm running pg_dump!</p>
  
  <pre><code>$ pg_lsclusters 
  Ver Cluster Port Status Owner    Data directory           Log file
  10  main    5432 online postgres /data/postgresql/10/main /var/log/postgresql/postgresql-10-main.log
  </code></pre>
  
  <p>I am deeply perplexed and hamstrung at present by this. Would deeply appreciate forward moving clues. If I am compelled to fish around in the dark I will probbaly uninstall postgres 11 and 12 again and see if that helps, else I will end up having to trace <code>/usr/share/postgresql-common/pg_wrapper</code> to see how and where the two paths of pg_dump and pg_restore diverge down incompatible version paths. </p>
  
  <p><strong>Update:</strong></p>
  
  <p>A further clue I have uncovered, that permits me a workaround but simply deepens the mystery is as follows:</p>
  
  <pre><code>$ sudo -u postgres pg_dump --host=live.lan --port=5432 --dbname=mydb --username=myuser --format=custom &gt; test.backup
  $ sudo -u postgres /usr/lib/postgresql/10/bin/pg_dump --host=live.lan --port=5432 --dbname=mydb --username=myuser --format=custom &gt; test2.backup
  $ sudo -u postgres pg_restore -l test.backup
  pg_restore: [archiver] unsupported version (1.14) in file header
  $ sudo -u postgres pg_restore -l test2.backup
  ... produces listing of contents ...
  $ sudo -u postgres pg_dump --version
  pg_dump (PostgreSQL) 10.11 (Ubuntu 10.11-1.pgdg18.04+1)
  $ sudo -u postgres /usr/lib/postgresql/10/bin/pg_dump --version
  pg_dump (PostgreSQL) 10.11 (Ubuntu 10.11-1.pgdg18.04+1)
  </code></pre>
  
  <p>That is perplexing beyond belief. The only possible explanations:</p>
  
  <ol>
  <li>in spite of reporting identical version numbers the two pg_dumps are different. I would rule this out as beyond belief.</li>
  <li>pg_dump runs pg_wrapper which runs /usr/lib/postgresql/10/bin/pg_dump with some mystery argument(s) that break it! </li>
  </ol>
  
  <p>The second is plausible, and will require me to instrument pg_wrapper to diagnose. </p>
  
  <p><strong>Update 2:</strong></p>
  
  <p>And one instrumentation of pg_wrapper later. It eventuates that pg_dump runs pg_wrapper which runs <code>/usr/lib/postgresql/12/bin/pg_dump</code> yet it runs <code>/usr/lib/postgresql/10/bin/pg_restore</code> ... Go figure! Starting to think this a postgresql version interoperability bug!</p>
  
  <p><strong>Update 3:</strong></p>
  
  <p>Looking deeper into <code>pg_wrapper</code>, nailed the cause, and yes I'd argue it's a pg_wrapper bug of a kind though it may be debatable, hardly though IMHO. Here's what it does:</p>
  
  <p>If <code>--host</code> is provided then it uses the newest version of postgresql installed (12 in my case and this is for pg_dump, so pg_dump 12 creates the dump)</p>
  
  <p>If <code>--host</code> is not provided then it consults the user config (10 in my case, and this is for pg_restore, so pg_restore 10 is run and it can't read a file created by pg_dump 12).</p>
  
  <p>So why is this a bug? Because I have a use config, and I'd like it respected whether or not I'm talking to a remote host. More to the point if I specify a host I certainly don't expect to use the latest local version ignoring local config. I'd expect either to respect local config (as is the case when no remote host is specified) or try to match the rmeote hosts version. Arbitrarily leaning on the latest installed version is IMHO deeply questionable.</p>
  
  <p>BUT it turns out there is a workaround that works. Essentially instead of:</p>
  
  <pre><code>sudo -u postgres pg_restore -l test.backup
  </code></pre>
  
  <p>this works:</p>
  
  <pre><code>sudo -u postgres pg_restore --host=localhost -l test.backup
  </code></pre>
  
  <p>By specifying the host ironically we force it to ignore local configs and use the newest version of pg_restore which seems to work fine restoring to a PG 10 cluster. </p>