{'fraction_non_alphanumeric': 0.019785937606974145, 'fraction_numerical': 0.0056368242086766015, 'mean_word_length': 4.419243136284937, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 15, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
CDTP Chain Distributed Transfer Protocol

The rapid growth of the internet in general and of bandwidth capacity at internet clients in particular poses increasing computation and bandwidth demands on internet servers. Internet access technologies like ADSL [DSL], Cable Modem and Wireless modem allow internet clients to access the internet with orders of magnitude more bandwidth than using traditional modems. We present CDTP a distributed transfer protocol that allows clients to cooperate and therefore remove the strain from the internet server thus achieving much better performance than traditional transfer protocols (e.g. FTP [FTP]). The CDTP server and client tools are presented also as well as results of experiments. Finally a bandwidth measurement technique is presented. CDTP tools use this technique to differentiate between slow and fast clients.


Abstract
The rapid growth of the internet in general and of bandwidth capacity at internet clients in particular poses increasing computation and bandwidth demands on internet servers. Internet access technologies like ADSL [DSL], Cable Modem and Wireless modem allow internet clients to access the internet with orders of magnitude more bandwidth than using traditional modems. We present CDTP a distributed transfer protocol that allows clients to cooperate and therefore remove the strain from the internet server thus achieving much better performance than traditional transfer protocols (e.g. FTP [FTP]). The CDTP server and client tools are presented also as well as results of experiments. Finally a bandwidth measurement technique is presented. CDTP tools use this technique to differentiate between slow and fast clients.

Introduction
As the internet becomes a standard source of information for increasing number of users, access technologies emerge that allow users faster access to the internet. Technologies like ADSL, Cable modem and Wireless modem allow users access rates of several Megabits per second instead of the 56Kbps allowed by the traditional modems. These access techniques and the increasing number of internet users pose increasing bandwidth and computational demands on internet servers since they have to support both increasing numbers of clients and to be able to supply data at much higher rate to each of the clients. Bandwidth and computational power demands of traditional transfer protocols like FTP grow linearly with the number of users and their bandwidth capacity. Various techniques were developed to solve this problem: Server farms with load balancers enable distributing the load between several physical servers at the site. Such solutions are more fault tolerant than just using a single server but the required bandwidth and the total computational power is still proportional to the number of users and their access speeds. Mirror site arrays like Akamai [AKA] are another way to solve the problem, this solution has the robustness advantage of a load balanced server farm and in addition it allows clients to be served by nearby servers, still from the point of view of the provider the amount of computing power and bandwidth that has to be bought is proportional to the number of users and their speeds.
Another way is to use distributed transfer protocols where clients pass data to other clients. Napster [NAP] and Gnutella [GNU] are examples for such techniques, in both cases clients are expected to store data that they download and to be ready to serve it to other clients that need the data. The problem here is that internet clients that participate in such a network may delete the downloaded data at any time, they are not even guaranteed to be constantly connected to the internet, these facts restrict the usability of such solutions to well known files like music or movie files since 1: The number of such different files is limited and 2: clients tend to store such files for long periods of time. In this work a new type of distributed protocol is presented it is called CDTP and like Napster and Gnutella it makes internet clients act as servers and transmit data to each other thus relaxing bandwidth demands from the server. Unlike Napster and Gnutella CDTP does not require clients to be constantly logged on to the internet or to store downloaded files for long periods of time, in fact clients do not have to store downloaded files at all, since the only time a client has to serve other clients is when it is downloading files itself. The above relaxation of requirements is based on the observation that as long as clients do not access the server simultaneously the server has no trouble serving them. When a large number clients access a server at the same time (Like in the case when a company like Microsoft releases a new patch or when a record company releases a new single of a known rock band), then the server may have trouble to service all the clients unless enough processing power and bandwidth was prepared in advance. When such situation arises CDTP clients start cooperating and pass the requested data to each other and in this way reduce the load from the server. To accomplish the above task efficiently it is important that clients will be served by clients with close enough bandwidth capacity, otherwise fast clients will be slowed down by slow clients. A packet pair based bandwidth comparison method is presented, this method measures the capacity of the bottleneck link between a source-destination pair, and it is optimised to the CDTP requirements.
The rest of this paper is structured as follows: Chapter 1 presents existing distributed transfer protocols, chapter 2 presents CDTP, chapter 3 presents existing bandwidth measurement techniques, and the bandwidth measurement technique used by CDTP. chapter 4 describes CDTP in detail, chapter 5 presents results of experiments with CDTP and finally chapter 6 concludes and suggests future directions for research. The exact CDTP specification resides in appendix A. A proof of the packet pair property under the model that is defined in this paper is given in appendix B.

Traditional transfer protocols
The traditional and most widely used file transfer and sharing technology is the FTP protocol. In FTP there is a server that holds all the files and clients that can access the server and perform actions for querying the server's contents and getting the files. The disadvantage of FTP is that it is centralized and in order to serve a large number of clients the server should reside on a powerful computer and have a lot of available bandwidth.
Another related traditional technology is IP Multicast [COM]. IP Multicast is not a parallel technology to FTP (Layer 3 in the OSI Model as opposed to Layer 7), IP Multicast enables hosts to form groups where each member in the group can send IP packets that will reach all the groups members. This could be a basis for an efficient transfer protocol because once a group is formed the server can send packets to all of the group and thus conserve bandwidth and resources. The main disadvantage with using IP Multicast for the above purpose is that all the clients should request the file simultaneously, otherwise the server should resend the data and nothing is gained.

Distributed transfer protocols
In recent years file sharing technologies have emerged, these technologies form a virtual network (either centralized or decentralized) that enable users over the internet to share files and this way eliminating the need of a single server to hold all the files.
Napster is an example for a centralized file sharing technology. In Napster a single server holds an index of all the available files and their locations over the internet, when a client wishes to download a file, it access the server with a request and the server directs him to another client that stores the file.
Gnutella is an example for a decentralized file sharing technology. In Gnutella there is no server, instead the hosts in the network that called servants (because they act as both clients and servers) exchange information between themselves about the whereabouts of files in the network using a routing like protocol.
Both file sharing techniques are great for exchanging music and video files between peers over the internet but are less suitable for servers. In Gnutella there is a performance problem because there is no centralized server and therefore clients has to search for the files they need all over the internet. In both Gnutella and Napster files should be kept by clients even after they are no longer needed, which can be reasonable for music files, but unreasonable for downloaded software patches. Also both technologies are not suitable for files that change frequently.
CDTP is a breed between the traditional data transfer methods and the new file sharing methods. It's functionality is similar to FTP and it's implementation is similar to Napster, a major difference between CDTP and Napster is that files are stored in clients only during the time they are being received thus eliminating the two limitations described above.

CDTP at a glance
In the CDTP network there is a single server and multiple clients. The role of the server is to store and transmit files to the clients and to coordinate between clients that serve each other as will be described below. The clients request files from the server and while the file is being received they feed it to other clients that need the same file. To service multiple clients that request the same file the server creates chains of clients that service each other. That is the server feeds the file to the first client in the chain which in turn feeds it to the second and so forth. Each chain is identified by the bandwidth between the server and the clients in the chain. See Figure 2 -CDTP Chains. When a new client requests a file the server first finds an appropriate chain to attach the client to. Essentially the server looks for a chain whose bandwidth is similar to the new client bandwidth and whose end node has not progressed too far in the reception. The chain is selected and the client is serviced in the following way: • The server measures the bandwidth between itself and the client.
• The server selects a chain whose bandwidth is compatible to the bandwidth that was measured in the previous step and whose last member has not received more than a predefined fraction of the file. If no such chain is found then the new client starts a new chain. • The server directs the client to the last client in the selected chain.
• The client requests the file from the last client in the selected chain.
• The last client in the selected chain measures the bandwidth between itself and the new client and if it is no less than the selected chain bandwidth then it begins feeding the new client with the requested file. If no compatible chain was found then a new chain is formed and the client is fed directly by the server. The server constantly keeps track on the progress of individual clients by receiving progress reports from them, this way when a chain is broken because one of the clients stopped servicing the client that after him in the chain, the client that stopped receiving data contacts the server with a retransmission request and the server is responsible for finding a new chain for the client with similar amount of progress. If no such chain is found then the server services the client directly by forming a new chain and supplying the rest of the file.

Bandwidth measurement
In this section a new bandwidth measurement method is presented. It combines two existing techniques, the packet pair technique ( described in 3.2.1) and delay based techniques (described in 3.2.2), the new technique is presented in 3.3.

The Model
A typical network model is comprised of network hosts that generate and receive traffic, routers that transfer the network traffic and links that connect the above nodes. Internet

Figure 3 -Internet Path Model
The delay of a packet sent between hosts is comprised of the time it takes to send the packet over all of the links plus the sum of latencies of the links plus the sum of queuing delays at the hosts and routers due to processing time and cross traffic. Cross traffic delay is defined as the time a packet is delayed inside a node while waiting for the node to transmit other packets over the next link along it's path . Denote S as the size of the packet, b 1 ,..,b n the link capacities along the path, l 1 ,..,l n the link latencies and q 1 ,..,q n the queuing delays at nodes along it's path (note then q n+1 equals 0). We then get that the time it takes for the packet to reach it's destination is: Two quantities are interesting when measuring bandwidth of a network path: The path capacity bandwidth (PCB) which is defined as Min i =1,..,n (b i ) and the path available bandwidth (PAB) which is defined the as Min i =1,..,n (b i -c i ) where c i is the cross traffic capacity over the nodes i and i+1. In this work the former is investigated.

Existing measurement techniques
Two categories of BW measurement exist. One measures PCB (see section 3.2.1) and is based on the packet pair property of network paths that was first identified by [JAC1] . The other measures all the link capacities along a network path (see section 3.2.2) and is based on comparing delays to different nodes along the path. This technique was first introduced by [JAC2].

Packet Pair based techniques
The packet pair property states that if two packets of the same size are sent back to back then the difference between their arrival times is equal to the propagation delay of a packet over the bottleneck link. More formally define t i j to be the arrival time of the jth packet at node i then t n 1 -t n 0 = max i (S/b i ). And therefore PCB = S/(t n 1 -t n 0 ). A detailed proof of the property can be found in [KES]. The reasoning is that the two packets will queue at the bottleneck link and at no other link afterwards and therefore will arrive to the destination with the same spacing as they left the bottleneck link (See Figure 4).

Figure 4 -Packet Pair Property
The correctness of the above property depends on several assumptions: • The routers on the path are store and forward, meaning that a router begins processing a packet only after it's last bit has been received. This property is the common implementation of routers on the internet.
• The two packets are sent sufficiently close to one another. This assumption may be violated only when trying to measure extremely fast links which are not the focus of this paper. • Both packets take the same route to the destination. It was shown by [PAX] that this assumption is usually true because routes are changed infrequently (the scale is hours).
• There are no multi-channel links. If a link between two nodes has multiple channels then the first and second packets may be sent on different links and this way they may be reordered or their spacing may be changed. Most links are not multi-channel, the exceptions are very fast ATM links and two channel ISDN links.
• There is no cross traffic. Cross traffic may corrupt the results of bandwidth measurement using packet pair. If a cross traffic packet is queued between the two packets of the pair after the bottleneck link then the reported BW will be lower than the actual bottleneck bandwidth. If a cross traffic packet is queued before the first packet in the pair after the bottleneck link then the reported BW will be higher than the actual bottleneck bandwidth. This assumption is usually not true because internet links share traffic from many hosts, therefore various filtering techniques are used to filter out the measurements that were affected by cross traffic. Below is a summary of related work on the packet pair technique: The packet pair property was first identified by [JAC1]. [KES] used the packet pair property to measure PAB under fair queuing assumption in the routers along the path. Under fair queuing implementation the queuing delay at the routers is proportional to the size of the packet and therefore as long as the cross traffic is stable, the available bandwidth at the bottleneck link can be measured correctly. Fair queuing is not the common implementation in routers, routers use the simpler FCFS (First come first serve) implementation. [CAR] presented tools to measure both PCB (bprobe) and PAB (cprobe). bprobe sends pairs of echo [POS] packets to the destination host and measures the difference of arrival times in each pair. The results are then collected and the ones that were affected by cross traffic are filtered out using histograms. The advantage of this technique is that it does not require any cooperation from the receiver host. The disadvantage is that the path must be symmetric, otherwise the reported PCB may belong to a link in the opposite direction. Another problem is the relying on control packets that are not always forwarded in a timely manner by the routers along the path. [PAX] Solved the problem of links that has multiple channels by sending a bunch of packets instead of only two (Hence the name of the method PBM -packet punch modes). In PBM different bunch sizes where used and the analysis of the gathered results allowed multi-modal distribution so that both multiple channel situations and bottleneck changes over time could be identified. [DOV] Showed that because of cross traffic, the distribution of packet pair results is multi-modal and that under heavy cross traffic the capacity mode is not the dominant mode. The packet size was also investigated and it was shown that although bigger packets are more immune to noise they also produce more noise because the bigger the packets, the bigger is the chance that cross traffic will be queued between the first and second packet in the pair. It was also shown that using trains of packets that are long enough produce a single modal distribution of results that converge to a value called ADR (Asymptotic dispersion rate) which is smaller than the bottleneck link capacity. A tool called pathrate was introduced that first found the different distribution modes using packet pair and then found ADR using packet trains, finally it selected the capacity mode from the first stage that was the minimal mode that is still bigger than ADR, this mode was selected as the capacity mode. It should be noted that the last step is a heuristic. [LAI1] Measures BW passively. Instead of actively sending packet pairs probes are placed on network hosts. The probes identify packets that came from the same source and record their spacing and then send it to a server that groups and analyses the results using density functions. In addition to the regular cross traffic effects that should be filtered out, the situation when the pair packets where not spaced close enough to each other can occur in passive measurement and therefore is identified and filtered out.

Delay based techniques
Delay based techniques measure the capacities of all of the links along the path by sending echo packets with limited TTL field. The filtering of the results of delay based techniques is easier because it is enough to take the minimum of each sample set. The disadvantages are that a large amount of packets has to be sent (Because there is at least one set of measurements for each node along the path), hidden hops may distort the results, and noise may be amplified because of arithmetic operations that must be performed between measurement results that may be inaccurate.
Below is a summary of related work on delay based techniques: [JAC2] Presents pathchar, a tool that sends a set of packets to a destination node, each with different TTL from 1 to n. This process is repeated with different packet sizes so that finally and a set of graphs is produced each graph shows the delays to a node for different packet sizes. Subtracting a graph of node k from the graph of node k+1 and taking the inverse of the slope of the resulting graph yields the capacity of the link between nodes k and k+1. Filtering out cross traffic affected measurements is done by taking the minimum of each sample set. [DOW] Shows how the performance of pathchar can be improved in terms of the number of measurements performed. The basic idea is to use adaptive data collection that is to stop probing nodes once their delay value converges. [JIA] Presents a variation of the pathchar idea that enables to measure links BW in both directions of the path. [LAI2] Introduces a new technique called Packet Tailgating. Two steps are performed, the first is similar to pathchar's last step where the sum of bandwidths between the source and destination hops is calculated using linear regression on several packet sizes. The second step performs n measurements in which a very small packet is queued behind a very large packet until the large packet is dropped by one of the routers along the path due to a limited TTL. The second step is performed several times for each TTL value between 1 and n and the minimum value is taken from each sample set. The advantages of this technique are that less packets need to be sent since the second phase does not require linear regression and therefore there is no need to send packets of different sizes, also ICMP packets are not used (Neither echo nor TTL exceeded), and therefore the technique does not depend on timely processing of such packets in routers.

Bandwidth measurement in CDTP
The Main requirement in CDTP is that the BW measurement must be quick, because it has to be performed each time a new client requests a file. As a result delay based techniques are inadequate because they require to send packets to each node along the path which can consists of more the 15 hops. Packet pair methods are more suitable because packets are sent to the destination host only , the problem with packet pair is that it is very sensitive to noise along the path and to filter out noise a large number of measurements has to be taken. Our technique uses a packet pair like method, but it's filtering method is the same as in the delay based techniques that is taking a minimum of each sample set is enough.
The technique is made of m identical steps each step is comprised of four stages. The bandwidth calculation is performed on the minimum value of each stage at the end of all steps.
• Stage one of each step sends a packet of size S to the peer host. Upon receiving the packet the peer host sends a reply packet of size S. The sender host calculates the time it took the reply packet to arrive and stores the result in T 1 i variable where i is the index of the step. • q * -The queuing delay at the bottleneck link.
Below will be shown how to calculate b * . It should be noted that due to the non deterministic nature of the Internet the values of q i are different for each measurement and therefore the values of T i also change with each measurement. In the below calculations it is assumed that the q i values are constant. The effect of this assumption is minimized by the filtering method that minimizes T i j and therefore minimizes the differences between the q i of the selected measurements.
Using the fact that in all of the stages the reply is the same size and under the above assumption we get: By the packet pair property (proved in Appendix B-Packet Pair Proof) we get: Subtracting (3) from (4) and reordering yields: Formula (5) gives the bottleneck link capacity but unfortunately none of the T i ' can be measured directly.
From the definitions we get: Now by substituting T i ' with T i -T i~ in (5) and by taking (2) into account we get : Formula (7) gives the bottleneck link bandwidth as a function of T i which is measured directly.
The queuing delay at the bottleneck can be derived similarly: By multiplying (3) by 2, subtracting (4) and reordering we get: By substituting T i ' with T i -T i~ in (8) and by taking (2) into account we get : The above method yields the bottleneck link bandwidth by taking the minimum result for each stage and therefore requires much less samples than other packet pair techniques. It also does not require to build up a sample set for each node along the path as required by the delay based techniques.
To avoid measurement errors due to modem compression random bits are sent as payload in the measurement packets.
The above method still cannot measure correctly paths whose bottleneck link has multichannels.

Software entities and their roles
CDTP is a session oriented protocol. Each dialog between CDTP client and CDTP server belongs to a certain session with unique ID. Each sub protocol of CDTP defines a set of session types: The Query sub protocol has one session type which is the query session, the Bandwidth sub protocol has also one session type which is the BW session, and the CT protocol has 3 session types, the transfer session the chain session and the file session. The two later session types are internal to the server and their ID is never passed by the protocol.
• Query session is opened between client and server as soon as the client performs successful START_QUERY_SESSION request and remains until the client disconnects either by STOP_QUERY_SESSION command or because of a time out. The number of active Query sessions in the server is exactly the same as the number of active CDTP clients. • Bandwidth session is opened between server and client or between service client and client when server or service client needs to measure the bandwidth between himself and the client.
• Transfer session is opened between server and client when the client initiates a get request, the transfer session remains until the completion of the get request.
• File session is opened when a is file requested for the first time by any client, the file session is closed after the last client that was requesting the file was serviced.
• Chain Session is opened either by a client that is the first in the file session or by a client that did not find any suitable chain in the file session (suitability of a chain depends on the amount of BW between server and client compared to the amount of BW in the chain, and the amount of data that client at the tail of the chain received compared to the amount of data the requesting client received.

The Protocol
CDTP consists of three sub protocols: Query, Bandwidth and CT.
The Query sub protocol is a subset of FTP commands, including Ls, Pwd, Cd, and Get. All commands except for the Get reply are trivial. Get Reply passes to the client all necessary data to contact the service client which includes: The IP address of the service client, the TCP transfer port at the service client, the transfer session Id of the service client (will be passed to the service client as part of the transfer request for authentication), a transfer session id for the client's current transfer session, the size of the requested file and the BW that the server measured between itself and the client.
The Bandwidth sub protocol is used to measure the bandwidth between the server and client or between the server and a service client, and it is based on the technique described in 3.3.
The CT sub protocol defines the dialog between the client and service client. The client initiates a transfer request from the service client the service client measures the BW between itself and the client and if it is compatible to the BW between the server and the client then the client is serviced. The protocol commands are: • Start Download Request -This is the first request that is sent by the client after the TCP session is established. It's parameters are: The Service client's transfer session id (For authentication), the client's transfer session id (will be passed in any subsequent message of the protocol), the position in the file to start transfer from, the BW between the client and the server (to be compared by the service client to the BW between himself and the client), and the number of the client's port for BW measurement replies. • Transfer Reply -These periodic replies are sent from the service client to the client over the established TCP session. Each reply contains some control parameters and a chunk of the requested file. All reply chunks are of the same size and the last reply chunk is padded with zeros if needed. The parameters are: The transfer session Id, the size of the actual chunk of the file, an indication if this chunk is the last one and the number of the current chunk.
• Transfer Denied reply -Can be sent by the service client over the TCP session at any time during the session it's parameters are: Transfer session id and the denial reason. The possible reasons are: Incompatible link speed or service client transfer session id, service client already serving someone or service client going down.
• Retransmission Request -Sent over UDP by the client to the server if connection to the service client could not be established or was broken. It's parameters are: the client's transfer session id, the number of bytes received by the client so far and an indication if the service client was engaged, this indication is needed so that the server will know if it can direct another client to the service client.
• Retransmission Reply -Sent over UDP by the server to the client to indicate an alternative service client or to indicate a failure, it's parameters are similar to the parameters of the get reply in the query sub protocol.
• Progress Report -Sent over UDP by the client to the server after each successful processing of a transfer reply from the service client, and carries the amount of data that the client received until that point. The server uses the received data when it decides which chain a new client should be assigned to.
• Disengage Command -Sent over UDP from client to the server to indicate that the transfer session has ended and no clients will be served over the session (if the client was the last in the chain then the chain is closed). The passed parameters are the amount of data received by the client and an indication if the service client was engaged by the client.

Acronyms in the Diagrams
• A solid line means a message between different hosts. • A dotted line means a command inside a host. • S Query -Component responsible for the query sub protocol in the server. • S Control -Component responsible for the control messages of the CT sub protocol in the server. • S Transfer -Component responsible for the data transfer messages of the CT sub protocol in the server. • S BW -Component responsible for the BW sub protocol in the server.
• C Client -Client that requests data.. • C Service -Client that serves another client.

First Client Flow
The figure below shows a flow for a client that requests a file, and a new chain has to be created for that client: 1. The client issues a get request to the server. 2. The Server measures the BW between himself and the client and creates a chain with the client as the head of the chain. 3. The Server replies the Get command where it specifies it's own address as the address to which the client should connect to receive the file it requested. 4. Upon receiving the reply, the client starts requesting the file from the server (the file is requested in chunks so that other client's could be served by the first client). During data retrieval the client sends progress reports to the server. 5. At the end the client issues a disengage command to the server and if no clients were added to the chain, then the server removes the chain.

General Client flow
The figure below shows a flow for a client that requests a file that already being transferred to other clients: 1. The client issues a get request to the server. 2. The Server measures the BW between himself and the client, finds a chain with compatible link speed and places the client on the tail of the found chain. 3. The Server replies the Get command where it specifies the address of the last client in the found chain as the address to which the client should connect to receive the file it requested. 4. Upon receiving the reply from the server the client requests the service client (whose address and port were given to the client in the get reply), to pass him the file. 5. The service client measures the BW between himself and the requesting client, sees that it compatible to his own and then starts sending data to the client. 6. The rest is like in the previous case.  1 -A transfer to a client was broken either because of an explicit message from the service client or because of a timeout. 2 -The client issues a retransmission request to the server. 3 -The server finds a suitable chain for the client (The chain is chosen according to two parameters: 1 -The BW to the client 2-The amount of data the client already received). 4 -A redirection reply is sent to the client,. 5 -The rest is similar to the last two scenarios.

CDTP experiments
The experiment was conducted in a lab with one computer that acted as a server and 3 other computers that acted as clients. The clients resided on the same 100MB Ethernet and therefore the access speed between them was high. The server was behind a wireless modem that was configured with 120Kbps, 240Kbps and 480Kbps speeds. In each of the tests a file was requested by all of the clients almost simultaneously (There was a 5 seconds difference between the requests. The requests were always been in the same order: Client 1 , client 2 and Client 3.) using CDTP and using FTP.

-CDTP vs FTP
In all of the CDTP experiments client 2 downloaded the file faster than client 1 and client 3 was faster than client 2. The reason is that client 2 started the downloaded 5 seconds after client 1 and client 3 started the download 5 seconds after client 2 and therefore part of the data that clients 2 and 3 were using was already cashed at clients 1 and 2 respectively.
In all of the FTP experiments client 2 was the slowest. The reason is that in FTP all 3 clients got the file directly from the server, and client 2 download was overlapped by either client 1 or client 3 or both during it's entire download period.
The bandwidth utilisation in CDTP was not optimal because of the bandwidth measurement phase at the beginning of the query. The utilisation was better in the last experiment (68% in the first, 66% in the second and 86% in the third) because the transferred file was larger and therefore the effect of the bandwidth measurement which is constant in size was smaller.

Bandwidth measurement experiments
The bandwidth measurement was tested both in lab conditions using a wireless modem and in internet conditions using a standard modem.
In the lab a path of 2 hops was measured: The first hop was a 100 Mbps fast Ethernet, the second hop was the bottleneck hop and it was tested with bandwidth of 60Kbps, 120 Kbps, 240 Kbps and 480 Kbps. The base packet size was 600 bytes and the number of steps per each measurement was 3.
The As can be seen in table 2 the measurement was very accurate except for one measurement with modem speed 120 kbps. The reason is probably a burst of cross traffic that disrupted the measurement. Taking a larger number of steps should have fixed the inaccuracy.
In the internet experiments the following setups were used: Setup 1 -Two PCs were both connected with a 38.4Kbps modem link to each other over a path of 12 hops. Setup 2 -Two PCs were connected over 16 hops. The first PC that was the sender was connected using a 38.4Kbps modem and the second that was the receiver was connected using a 384Kbps fractional E1 Link. Setup 3 -The same as setup two but with the roles of the sender and receiver reversed. In the reverse direction the number of hops was 15.
The number of hops was calculated using the trace-route utility in the PC. The actual link speed of the modem connections was not exactly known so it was approximated by correlating 3 sources: 1 -Pings to the nearest router 2 -An internet site that measures BW by downloading large amounts of data to the browser [BWP] 3 -Our own measurements. In all cases the measured BW (Both uplink and downlink) was around 26 kbps so it was used as the reference speed for the measurements.
Because of the internet instability the number of steps in each measurement was increased to 10. Therefore: And: Thus we get that the value in the third column in table 3 should be bigger than the value of the fifth column which in turn should be bigger than the value of the sixth column. The intuitive explanation of formula (11) is that the calculated bandwidth value should be greater when the delay is negated and the explanation for formula (12) is that when the packet is bigger than the delay inside the router has a lesser effect on the bandwidth calculation.
Now returning to measurements 2 and 5 in setup 1 it can be seen that formulas (7) and (8) do not hold and therefore the BW calculation is erroneous. A possible adaptive way to handle such situation is to continue measuring until the inequalities hold.

Summary and Future Research
In this work we presented CDTP, a distributed transfer protocol that uses the unused portion of the internet clients bandwidth to relax the computation and bandwidth requirements of internet servers and unlike Napster and Gnutella CDTP does not require that clients store the downloaded files for periods longer then necessary to receive them. A bandwidth measurement technique was presented that combined the ideas of the packet pair techniques with the ideas of delay based techniques to achieve accurate bandwidth measurements while using small amount of measurement packets. A way to extend CDTP is to use its ideas to transfer other kinds of traffic like HTTP or streaming multimedia. The main obstacle with HTTP is that it's traffic consists of much smaller data units and therefore the bandwidth measurement methods should be more efficient.
Another improvement could be to add the ability to detect multi-path bottleneck links and still send a small amount of bandwidth measurement packets.

7.
Appendix A -The Protocol • Description: A request to start a query session, the passed port numbers: • Client UDP Query Port -Will be used by the server to communicate to the client throughout the session of the Query protocol. • Service Client TCP Transfer Port: With this port the client acts as service client and serves other clients.
• Client UDP Control Port -Will be used by the server to communicate to the client throughout a session of Control Protocol.
• Port of the client's Peer 2 BW Protocol.
• Description: A reply to a start query session request by a client. If request is granted then a session Id is returned.
• Description: An indication that the query session is closed by one of the sides.  • Operation: PWD REPLY • Initiator: Server.
• Description: A request to start transferring the data.
• The start position indicates from what location in the file to start the transfer (It is needed when clients reassigned to alternate server after completing part of the transfer). • Chunk size indicates the amount of data sent with each transfer. The requested link speed is also provided, so that the Service Client will measure the link speed between itself and the client and if it is not within acceptable range, the request will be denied.
• Field 5 -Data Chunk: Field Size Same as "Chunk size" field indicated in START TANSFER REQUEST.
• Description: A transfer of requested data.
• All the chunks are of the size indicated in the request packet, When less data than Chunk size is passed (For example the last chunk or if the server does not have all the data yet ) the remainder is padded with zeros, the size of actual data is indicated in the actual data field. • The is last field indicates if this transfer is the last.
• TANSFER DENIED REPLY • Protocol: TCP. Description: The link between the client and the Service Client, is broken, or cannot be established therefore the client requests the server to give it an alternate source. Size received indicates the amount of data received by the client so far. The Service Client occupied field indicates if the service client has accepted the connection attempt from the client or not (It is important for the server to know this because it should not assign new clients to the service client's chain if it accepted the connection).
• Field 3 -IP of alternate service client. Field Size: 4 bytes.
• Field 4 -Port number of the TCP transfer port of the service client. Field Size 2 bytes.
• Field 5 -New Service Client Transfer Session Id. Field Size: 4 bytes.
Description: A reply from the server that indicates the alternate server to that will supply the rest of the data for the client, a new chain is used because the alternate server will be located in the new chain (The old one is broken) and therefore a new Service Client Transfer Session Id is given. The transfer session Id remains the same. Description: A report indicating the number of bytes received by the client so far. This info will be used by the server when it decides where to redirect clients that need retransmission, and where to assign new clients.
Description: An indication by the client that it is disengaging and therefore it's chain is closed. The No of bytes received is needed for statistics. The Service Client occupied field indicates if the service client has accepted the connection attempt from the client or not (It is important for the server to know this because it should not assign new clients to the service client's chain if it accepted the connection).

Bandwidth Protocol
The bandwidth protocol used by two peers to determine the speed of the links between them. The protocol works in the following way: 1 -Peer 1 sends "breakthrough" packet to peer 2. The reason of this packet is to update caches in the routers between the two peers. 2 -Peer 2 replies that it received the "breakthrough" packet. 3 -Peer 1 sends a packet to peer 2, before sending the packet peer 1 measures the current time and put it in the sent packet. 4 -Peer 2 sends the same packet exactly to Peer 1. 5 -When Peer 1 receives the reply it measures the current time, subtracts from it the time of sending (This time is written in the received packet) and stores the result as T1. 6 -Peer 1 sends a packet which is # times bigger than the previous one (# is the division factor), with an indication to peer 2 to divide the size of the returned packet by #. 7 -Peer 2 receives the packet, divides it's size by # and sends it back to peer1. 8 -When Peer 1 receives the reply packet it calculates the time the second packet travelled back and forth and stores it in T2. 9 -Peer 1 sends a pair of packets back to back, the packet sizes are the same as the size of the packet in stage 3. Before sending the first packet of the pair peer 1 measures the current time and puts it in the sent packets. 10 -Upon receiving the second packet of the pair peer 2 sends a single packet of the same size back to peer 1. 11 -When Peer 1 receives the reply it calculates the time the packets travelled back and forth and stores it in T3. 12 -Peer 1 sends a pair of packets back to back, the packet sizes are the # times bigger than the original packet. Before sending the first packet of the pair peer 1 measures the current time and puts it in the sent packets. 13 -Upon receiving the second packet of the pair peer 2 divides the packet's size by # and sends a single packet back to peer 1. 14 -When Peer 1 receives the reply it calculates the time the packets travelled back and forth and stores it in T4.
• The above procedure is performed several times and then the minimum of each T i is taken and the bottleneck bandwidth is calculated using formula (7).