title: kernel-based (Linux) data relay between two TCP sockets
body: <p>I wrote <strong>TCP relay server</strong> which works like peer-to-peer router (supernode).</p>
  
  <p>The simplest case are two opened sockets and data relay between them:</p>
  
  <p><strong>clientA &lt;---> server &lt;---> clientB</strong></p>
  
  <p>However the server have to serve about 2000 such A-B pairs, ie. 4000 sockets...</p>
  
  <p>There are two well known data stream relay implementations in <strong>userland</strong> (based on <em>socketA.recv() --> socketB.send()</em> and <em>socketB.recv() --> socketA.send()</em>):</p>
  
  <ul>
  <li>using of <em>select</em> / <em>poll</em> functions (non-blocking method)</li>
  <li>using of threads / forks (blocking method)</li>
  </ul>
  
  <p>I used threads so in the worst case the server creates 2*2000 threads! I had to limit stack size and it works but is it right solution?</p>
  
  <p>Core of my question:</p>
  
  <p><strong>Is there a way to avoid active data relaying between two sockets in userland?</strong></p>
  
  <p>It seems there is a passive way. For example I can create file descriptor from each socket, create two pipes and use dup2() - the same method like stdin/out redirecting. Then two threads are useless for data relay and can be finished/closed.
  <strong>The question is if the server should ever close sockets and pipes and how to know when the pipe is broken to log the fact?</strong></p>
  
  <p>I've also found "socket pairs" but I am not sure about it for my purpose.</p>
  
  <p><strong>What solution would you advice to off-load the userland and limit amount fo threads?</strong></p>
  
  <p><strong>Some extra explanations:</strong></p>
  
  <ul>
  <li>The server has defined static routing table (eg. ID_A with ID_B - paired identifiers). Client A connects to the server and sends ID_A. Then the server waits for client B. When A and B are paired (both sockets opened) the server starts the data relay.</li>
  <li>Clients are simple devices behind symmetric NAT therefore N2N protocol or NAT traversal techniques are too complex for them.</li>
  </ul>
  
  <p><strong>Thanks to Gerhard Rieger I have the hint:</strong></p>
  
  <blockquote>
    <p>I am aware of two kernel space ways to avoid read/write, recv/send in
    user space:</p>
    
    <ul>
    <li>sendfile</li>
    <li>splice</li>
    </ul>
    
    <p>Both have restrictions regarding type of file descriptor.</p>
    
    <p>dup2 will not help to do something in kernel, AFAIK.</p>
  </blockquote>
  
  <p>Man pages: <a href="http://linux.die.net/man/2/splice" rel="nofollow noreferrer">splice(2)</a> <a href="http://man7.org/linux/man-pages/man2/splice.2.html" rel="nofollow noreferrer">splice(2)</a> <a href="http://linux.die.net/man/2/vmsplice" rel="nofollow noreferrer">vmsplice(2)</a> <a href="http://linux.die.net/man/2/sendfile" rel="nofollow noreferrer">sendfile(2)</a> <a href="http://linux.die.net/man/2/tee" rel="nofollow noreferrer">tee(2)</a></p>
  
  <p>Related links:</p>
  
  <ul>
  <li><a href="https://stackoverflow.com/questions/8626263/understanding-sendfile-and-splice">Understanding sendfile() and splice()</a></li>
  <li><a href="http://blog.superpat.com/2010/06/01/zero-copy-in-linux-with-sendfile-and-splice/" rel="nofollow noreferrer">http://blog.superpat.com/2010/06/01/zero-copy-in-linux-with-sendfile-and-splice/</a></li>
  <li><a href="http://yarchive.net/comp/linux/splice.html" rel="nofollow noreferrer">http://yarchive.net/comp/linux/splice.html</a> (Linus)</li>
  <li><a href="https://stackoverflow.com/questions/13215656/c-sendfile-and-send-difference">C, sendfile() and send() difference?</a></li>
  <li><a href="https://stackoverflow.com/questions/15355069/bridging-between-two-file-descriptors">bridging between two file descriptors</a></li>
  <li><a href="https://stackoverflow.com/questions/2014033/send-and-receive-a-file-in-socket-programming-in-linux-with-c-c-gcc-g">Send and Receive a file in socket programming in Linux with C/C++ (GCC/G++)</a></li>
  <li><a href="http://ogris.de/howtos/splice.html" rel="nofollow noreferrer">http://ogris.de/howtos/splice.html</a></li>
  </ul>