5.4 Full Tunnel Overhead 6.2 Discovering Faults 6 Experiences 6.1 Deployments need to be as proactive about ensuring the connection like in IPOP and Hamachi. The important take-aways from this experiment are: 1) each additional member in a VPN requires more mem- ory, but the structured P2P approach places this bur- den only on the individual client and only when there is active communication; 2) maintaining NAT mappings through keep-alive messages (a ping every 15 seconds) requires very little bandwidth; and 3) all traffic routes through the OpenVPN server and it pays twice for each inter-client packet thus as inter-client communication in- creases OpenVPN server’s bandwidth becomes a bottle- neck. This experiment was done to evaluate the overhead introduced using our Ethernet packet based full tunnel technique. The environment for this experiment consists of a VPN client and gateway located at a residence and University, respectively. The gateway is configured using Linux with iptables’ masquerading. The client is also us- ing Linux. The test performed consist of calculating the average latency, through a minutes worth of ping mea- surements, between the client and Google as well as the gateway’s public and private addresses. As shown in Table 4, there is not a significant latency difference in the full tunnel approaches. One interest- ing result is the latency to gateways public address in the routing test, which is a result of the ping being sent inse- curely avoiding the VPN stack completely. tion, as part of the “Grid Appliance” project [40]. Over the years, the Grid Appliance has matured and so has our usage of IPOP. Currently, the Grid Appliance is used as the basis for a free-to-join computer architecture consist- ing of over 400 compute resources spanning 5 seed uni- versities with users across the United States and abroad. The Grid Appliance allows easy creation of ad hoc, dis- tributed grids through the use of the IPOP and the struc- tured P2P DHT. Experienced users can use the system to create large grids in less than an hour with most of the time spent waiting for virtual machines to boot. Non- expert users can quickly access the distributed grid in a matter of minutes by downloading a virtual machine manager and the Grid Appliance. The comment we re- ceive most often in our polls is “I was surprised at how it just works.” Motivation for a significant portion of our work lies in a desire to integrate other resources, not only vir- tual appliances, in the same “just works” manner. The GroupVPN described in this paper is the same software stack that runs inside Grid Appliance but also allows ser- vices the Grid Appliance to connect with external NFS and external grid resources. Our experience with P2P VPN software also includes SocialVPN [13]. The SocialVPN places each client in their own private network, where they are only address- able by their friends as defined by third party social net- work. Unlike GroupVPN, this makes each user the mas- ter of who does and does not have access to their re- sources. While SocialVPN targets the creation of “per- sonal” VPNs, which are often asymmetrical, it does not lend itself well to environments that require some form of central management and symmetric connectivity such as a computing grid or a small to medium businesses. In this section, we share our experiences in manag- ing and finding faults in a deployed P2P VPN overlay bootstrap service we run on Planet-Lab [8]. Planet-Lab serves as a challenging environment that stresses our sys- tem and helps us assess its reliability, stability, and con- sistency. The basic consistency check we use to discover faults is to compare peers knowledge of the pool and test each peer for congruence with both its first and second left and right neighbors. We call this a crawl. This in- formation is stored in a database, which we can later use to find nodes that have been inconsistent many consec- utive times. If a node is able to fix inconsistencies in future crawls, experience suggests the inconsistency was probably due to churn in the system. Otherwise, it will probably still be in an inconsistent state and we are able to query the node and other nodes nearby for additional state information. At a minimum, this would be help us determine if there exists a problem and potentially the stale state causing the connectivity issue. If this reveals Our work on the P2P VPN model presented in this pa- per is based on four years of work spent developing, de- ploying, and maintaining VN software for grid comput- ing [16, 41, 14]. In this section, we first present our dif- ferent deployments of our P2P VPN and then our experi- ences in debugging such a structured P2P VPN system. Initially IPOP was designed to assist in self- configuring ad hoc grid deployments using virtualiza- Table 4: Latency results comparing full tunnel approaches measured in ms. GW Pri is the gateways VPN address, GW Pub is the gateways public address, Ethernet was the Ethernet packet writing full tunnel method, routing was the routing ta- ble manipulation method, and none is the baseline without any VPN software. Google GW Pri GW Pub Ethernet 70.6 12.9 13.9 Routing 71.4 13.2 11.0 None 66.1 N/A 10.9 11