Bitcoin uses TCP for P2P but why is UDP not used? Modern day internet has relatively low packet loss rate so UDP is reliable. Even if some packets are dropped the peers can always request for the blocks. With TCP, the network is rather staticly constructed whereas UDP you can construct a random graph each time a block is propagated as its connectionless.

NANO uses UDP and I think it’s a good approach. I’m just curious as to if I’m missing something. Bootstrapping can obviously be done using TCP


The Bitcoin system doesn't have just one network protocol: Any way of obtaining the blocks is equally valid-- blocks over freenet, over satellite broadcast, over the P2P network, all work just as well and are used in practice. UDP is used with bitcoin too, by the Fibre protocol.

As far the common Bitcoin P2P protocol goes UDP is not an especially good fit for most operations that it performs. Bitcoin needs to reliably send messages far larger than an IP packet, like transactions and blocks. Although the internet is "reliable" packet loss rates of 1-3% are common. This means that any application needing to communicate large messages with UDP must implement packetization, retransmission, reordering, etc-- the same things that TCP already implements for us. Many applications that have 'rolled their own TCP' in userspace have ended up with exploitable bugs in them, so it isn't something that should be done without good cause.

UDP also has the issue of NAT traversal: Getting bidirectional communication across a NAT with UDP is not a simple matter. Crossing anything more complicated than a full cone nat requires considerable amounts of special code but without it there will be a many hosts that just can't talk to other peers with UDP.

You mention connectionlessness allowing you relay to a random host in the network, but in Bitcoin P2P we make use of the consistent relationship between nodes to make the network more reliable and efficient. Nodes have an idea what their peers already know and can avoid sending them redundant data. Nodes also know which peers have been fastest in sending them blocks in the past and handle them specially. Likewise, even if the protocol is connectionless there are costs to processing messages from peers by prioritizing handling messages from existing peers, Bitcoin reduces the impact of some kinds of DOS attack. Handling larger than one-packet messages, retransmissions, etc. also mean that even with a connectionless transport there still needs to be some kind of persistent state.

But an if implementation did want to randomly connect to send a message it could, at an extra cost is a couple round trips for the handshake. Or not even that: there are only about 10,000 reachable nodes in the P2P network, there wouldn't be much of a challenge in holding open a mostly idle TCP connection to each and ever one of them if all the state retained were just the TCP state. So for that point the use of UDP would at best just be an optimization for something that could already be done but isn't done. I think it would be more interesting to demonstrate the usefulness of having all those connections first, before worrying about optimizing it.

So why might UDP transport be useful to the Bitcoin P2P protocol?

  • To get lowest latency block transfer possible there must be no round trips even in the face of packet loss which precludes TCP and which is why Fibre uses UDP. But to get no round trips the protocol must be able to handle loss without retransmission and to get low latency a block must be decodable with a minimum amount of data received. This requires very sophisticated error correction techniques which are an evolving area and haven't become mature enough yet to consider mainstreaming them. The latency benefits of fibre also exist so long as a small number of hosts are using it, since it does the heavy lifting of taking blocks all around the world. Round trips don't cause much harm on low latency links. And this latency concern only applies to block relay.

  • Unlike TCP, UDP requires some amount of NAT traversal handling to simply get bidirectional communication working. But combined with full nat traversal handling UDP is often able to establish communication between hosts which are both behind different NATs. This could be helpful for the Bitcoin P2P network since most hosts are unreachable behind a nat. So, ironically, one of UDP's challenges is also one of its uses. However, to connect hosts which are both behind a net the assistance of a third party non-natted host is required along with even more NAT traversal code. Considering the complexity of traversal improving Bitcoin's support for TCP port mapping (e.g. implementing NAT-PMP) would probably be a better return on investment right now.

  • Use of UDP would allow for "worst than best effort" traffic handling. For low priority "bulk traffic" like syncing the blockchain it would be nice if the traffic carefully shaped itself to avoid interfering with other traffic on the network. Alternative congestion control algorithims like LEDBAT make it possible to transfer low priority data with minimal impact. But since TCP stacks don't yet commonly support these congestion control approaches, applications that want them currently need to implement their own transports. For Bitcoin it would be ideal if we could just flip a socket option and turn LEDBAT on and off on existing connections (e.g. when the peer is requesting historical blocks or in the future when something like Fibre is being used by the node to relay new blocks) but that isn't an option yet.

[These last two reasons are the reasons that Bittorrent usually uses UDP]

  • @g-maxwell Great answer. What do you think about the following: If its for a PoS system in which certain validators can process transactions only. Instead of relying on TCP block propagation, do you think it would be useful to have those validators accept UDP transaction requests so that we can get transactions validated faster? In other words, do you see any disadvantage to use TCP for block propagation (large data) and UDP for single transaction request so that the nodes do not need to establish TCP connection. – user2584960 Sep 13 '18 at 17:50
  • 1
    Why do transactions need to propagate fast? That premise is generally in conflict with privacy. – Pieter Wuille Sep 13 '18 at 18:02
  • Because if we want real adoption for crypto two things need to happen: stable currency and confirmation time. We can solve stablility via pegging and that leaves us with confirmation time. In order to achieve 2-3 second confirmation time we need faster relay mechnism than propagation. – user2584960 Sep 13 '18 at 19:37
  • 2
    Bitcoin achieves subsecond most of the network propagation time for small blocks... there is no need to use anything other than TCP for that. But 2-3 second true confirmation time is pushing with the laws of physics for an actually decentralized network that spans earth and is probably not practically possible with real networks. Sometimes though not often whole continents are mostly severed from each other for minutes. – G. Maxwell Sep 13 '18 at 19:53
  • If you can put the whole POW vs Pos argument aside I actually have a working prototype that has run a real test net in a geographically dislocated setting where I achieved (my team) a 6-7 second true confirmation. This is not including transaction submission time of course. So to ask the question here Greg: would UDP transaction submission be a good idea in my setting? Appreciate your insight as you are well respected in my book. – user2584960 Sep 13 '18 at 22:56

Simply, NANO uses UDP for blocks which are usually (certainly mostly, but always??) lower than the theoretical size limit of a UDP packet, 65 KB. In Bitcoin's case, transactions ≠ blocks. It's surely inefficient to transfer long data (Bitcoin blocks can be up to 4 megabytes - theoretically) using UDP, and that's also how other services choose between them. Besides, even if blocks were a few kilobytes, packet drop will cause miners to lose A LOT, as a few seconds delay is significant.

If the data is small, you can just send one packet (which is usually much lower than 65 KB). Otherwise, using TCP, handshake, keepalive things are not expensive.

Trivia: In Satoshi's client, there were no additional error correction codes in the protocol. Satoshi trusted TCP error correction.

  • Block is bitcoin is 1 mb. sure, it won't fit into one UDP datagram, and even if it did, at 65KB, it is still subject to fragmentation. However, it makes no sense to not allow the transaction themselves to be UDP based – user2584960 Sep 13 '18 at 17:41

UDP is not very useful in Bitcoin. It is largely a unidirectional communication structure, whereas Bitcoin relies on bidirectional communication. When a node connects to another node, there is saved state for those nodes (nodes keep track of things that they have sent to other nodes and of things they have received from other nodes) and there is back and forth communication (e.g. send inv, receive getdata response, and send the data). UDP is not useful for that, but TCP is.

Furthermore, UDP exposes the nodes to the unreliability of the networks. The packet loss rate is not 0 (and cannot be 0), so using UDP results in a lot of extra data being sent due to packet loss. This is because blocks and transactions cannot be missing any bytes whatsoever. Otherwise they will be invalid. With TCP, if a packet is dropped, TCP will handle the resending of the data. But with UDP that needs to be handled at the application layer and it just gets complicated since figuring out what has been dropped requires bidirectional communication which is not easily done with UDP.

So overall, UDP is not useful for Bitcoin. It does not support bidirectional communication and it does not guarantee delivery of packets. Both of those things are necessary for Bitcoin's P2P protocol to work, so UDP can't be used. TCP is does both of those things at the protocol level, so it is used.

  • This is incorrect. First of all UDP is not a unidirectional communication. UDP is connectionless, however statefulness can be implemented at the application layer, which BITCOIN already does. see their p2p system. Like I said, the loss of the packet isnt an issue as the whole idea of a p2p network is so that its tolerant to packet loss. So what if a node didn't get a transaction, just ask for it again. Your answer is not very helpful. sorry. – user2584960 Sep 13 '18 at 17:40
  • 1
    I also cringed a bit at the unidirectional claim, but it's not entirely untrue: that's why NAT traversal is an issue for UDP where it isn't one for TCP. It is the case the statefulness can be and is implemented in the application but the statefulness of TCP itself is really modest compared to the general statefulness of the protocol. As my answer pointed out: it would be totally reasonable for each bitcoin node to have a TCP connection up with most of the other reachable nodes. The state costs for the TCP itself aren't that substantial. The bitcoin protocol state is another matter... – G. Maxwell Sep 13 '18 at 21:22

There are several Reliable UDP implementations available by now, so requirements like packet loss recovery and statefulness can be addressed. One example is the UDT library developed by the University of Chicago: http://udt.sourceforge.net/

We've developed a protocol named SRT (Secure Reliable Transport), initially based on UDT and extended by adding bi-directional data transfer and encryption to the mix. Although it's optimized for real-time data like video, it's content independent by design: https://github.com/Haivision/srt

In video use cases, we are seeing dramatically increased bandwidth utilization numbers using UDP over TCP (e.g. RTMP), especially relevant for longer distance communication. I do believe it does make sense to evaluate UDP based alternatives for data transfer between nodes in order to increase network efficiency.

  • do you have contact info I can reach you at - I would like to discuss. – user2584960 Sep 27 '18 at 18:59

Your Answer

