Merge branch 'master' of https://git.erp5.org/repos/vifibnet

Conflicts:
	TODO
	tunnel.py
	vifibnet.py

README

+
+Note (for U): logs haven't changed. I just added some logs messages to notify
+              when a task has been finished
+
 Vifibnet is a daemon setting up a resilient virtual private network over the
 internet
 
 HOW TO:
-    Vifibnet ( sic ) has three separate components : a setup (setup.py), a 
+    Vifibnet ( sic ) has three separate components : a setup (setup.py), a
     server (registry.py) and a client (vifibnet.py.
     Lambda users only have to launch the setup and then their client.
     The server is meant to be started once on a node which also will be running
@@ -31,7 +35,7 @@ OPTIONS : REGISTRY.PY
             MUST contain the VPN network prefix in its serial number. To
             generate correct ca and key files for the 2001:db8:42:: prefix,
             the following command can be used :
-            openssl req -nodes -new -x509 -key ca.key -set_serial
+            openssl req -nodes -new -x509 -key ca.key -set_serial \
                     0x120010db80042 -days 365 -out ca.crt
 
     --key path
@@ -87,12 +91,11 @@ OPTIONS : VIFIBNET.PY
             Specify connection information to be advertised to other nodes.
             address MUST be a ipv4 address since as of now openvpn does not
             support ipv6 addresses.
-            proto should be either udp or tcp-client
+            Proto should be either udp or tcp-client
 
-    --internal-port port
-            Specify the port on which will be launched the openvpn server(s)
-            Can differ from port given in the --ip option.
-            Default : 1194
+    -i, --interface interface
+            Give one interface name for each use of the argument. The interface
+            will be used to detect other nodes on the local network.
 
     --peers-db-refresh duration
             Duration in seconds of the peers DB refresh interval.
@@ -138,17 +141,20 @@ OPTIONS : VIFIBNET.PY
             It takes between 3 times and 4 times the hello interval for babel
             to re-establish connection with a node for which the direct
             connection has been cut
-            Default : 30
+            Default : 15
 
     -w, --wireless
             Consider all interfaces as being wireless interfaces. Argument
             directly passed down to the babeld daemon
 
-    --proto p [p']
-            Protocol used by the openvpn server(s). Start one openvpn server
-            for each protocl specified.
-            p (and p') should be either udp or tcp-server
-            Default : udp
+    --pp port proto
+            Port and protocol used by the openvpn server(s). Start one openvpn
+            server for each couple port/protocol specified.
+            Additionally, if no external configuration is given in the command
+            line, vifibnet will attempt to forward a port with upnp for each
+            couple port/proto given.
+            Protocols should be either udp or tcp-server.
+            Default : (1194, udp)
 
     --tunnel-refresh duration
             Interval in seconds between two tunnel refresh. Refreshing tunnels
@@ -203,3 +209,5 @@ OPTIONS : VIFIBNET.PY
             The file should contain one option per line, possibly ommitting
             the '--'. Only long option are allowed (i.e "v 3" will not work
             while "verbose 3" will)
+            You can give a file ( with the @ prefix ) as an argument within a
+            file

TODO

-Bugs:
-    The address of the client is declared while it should only be the address
-    of the server
-
 To be done :
     use the server as a bootstrap node -> switch peer discovery to be done
     by vifibnet directly ?
@@ -17,14 +13,15 @@ To be done :
         provide false informations. Needs of signature
         If it is not sufficient, we could use avahi (dm-dns for linux)
 
-    Replace comments at the beginning of functions with docstrings & give all
-    fn docstrings
+    Write docstrings for all class/methods/functions
 
-    Upgrade the logging function in order to be able to log message like
-    "Refreshing peers DB ... done", or add log messages to specify that an
-    action advertised by a previous log message has been completed
 
 To be discussed:
+    G : There is a blacklist system now ( blacklisted prefixes are deleted from
+        the peers database ). Since all nodes whose packets are routed through
+        the local network are blacklisted, I think we should reset the blacklist
+        from time to time....
+
     U : Babel seems to be very long to establish the routes : maybe we should
         tell him thant we are not on a wired network but on a mobile network ?
     G : babel establish routes quickly enough i'd say. There are two new
@@ -52,3 +49,4 @@ To be discussed:
         reestablish connection, if a direct link is cut
     U : So we have to reduce the hello interval. 2min to detect a dead link is
         far too much.
+    G : k

db.py

@@ -15,28 +15,40 @@ class PeerManager:
         self._server_port = server_port
         self._db_size = db_size
         self._pp = pp
+        self._blacklist = [(prefix,)]
         self._manual = manual
 
         self._proxy = xmlrpclib.ServerProxy('http://%s:%u'
                 % (server, server_port))
 
-        utils.log('Connectiong to peers database', 4)
+        utils.log('Connectiong to peers database...', 4)
         self._db = sqlite3.connect(os.path.join(db_dir_path, 'peers.db'),
                                    isolation_level=None)
-        utils.log('Preparing peers database', 4)
+        utils.log('Database opened', 5)
+        utils.log('Preparing peers database...', 4)
         try:
             self._db.execute("UPDATE peers SET used = 0")
         except sqlite3.OperationalError, e:
             if e.args[0] == 'no such table: peers':
                 raise RuntimeError
+        utils.log('Database prepared', 5)
 
         self.next_refresh = time.time()
 
+    def reset_blacklist(self):
+        self._blacklist = [(self._prefix)]
+
+    def blacklist(self, prefix):
+        utils.log('Blacklisting %s' % (prefix,), 4)
+        self._db.execute("DELETE FROM peers WHERE prefix = ?", (prefix,))
+        self._blacklist = list(set(self._blacklist + [(prefix,)]))
+
     def refresh(self):
-        utils.log('Refreshing the peers DB', 2)
+        utils.log('Refreshing the peers DB...', 2)
         try:
             self._declare()
             self._populate()
+            utils.log('DB refreshed', 3)
             self.next_refresh = time.time() + self._refresh_time
         except socket.error, e:
             utils.log(str(e), 4)
@@ -45,14 +57,15 @@ class PeerManager:
 
     def _declare(self):
         if self._address != None:
-            utils.log('Sending connection info to server', 3)
+            utils.log('Sending connection info to server...', 3)
             self._proxy.declare((self._internal_ip,
                     utils.address_list(self._address)))
+            utils.log('Info sent', 5)
         else:
             utils.log("Warning : couldn't send ip, unknown external config", 4)
 
     def _populate(self):
-        utils.log('Populating the peers DB', 2)
+        utils.log('Populating the peers DB...', 2)
         new_peer_list = self._proxy.getPeerList(self._db_size,
                 self._internal_ip)
         self._db.execute("""DELETE FROM peers WHERE used <= 0 ORDER BY used,
@@ -61,7 +74,9 @@ class PeerManager:
                             (str(len(new_peer_list) - self._db_size),))
         self._db.executemany("""INSERT OR IGNORE INTO peers (prefix, address)
                                 VALUES (?,?)""", new_peer_list)
-        self._db.execute("DELETE FROM peers WHERE prefix = ?", (self._prefix,))
+        self._db.executemany("DELETE FROM peers WHERE prefix = ?",
+                              self._blacklist)
+        utils.log('DB populated', 3)
         utils.log('New peers : %s' % ', '.join(map(str, new_peer_list)), 5)
 
     def getUnusedPeers(self, peer_count):
@@ -73,16 +88,19 @@ class PeerManager:
         utils.log('Updating peers database : using peer ' + str(prefix), 5)
         self._db.execute("UPDATE peers SET used = 1 WHERE prefix = ?",
                 (prefix,))
+        utils.log('DB updated', 5)
 
     def unusePeer(self, prefix):
         utils.log('Updating peers database : unusing peer ' + str(prefix), 5)
         self._db.execute("UPDATE peers SET used = 0 WHERE prefix = ?",
                 (prefix,))
+        utils.log('DB updated', 5)
 
     def flagPeer(self, prefix):
         utils.log('Updating peers database : flagging peer ' + str(prefix), 5)
         self._db.execute("UPDATE peers SET used = -1 WHERE prefix = ?",
                 (prefix,))
+        utils.log('DB updated', 5)
 
     def handle_message(self, msg):
         script_type, arg = msg.split()

plib.py

@@ -18,7 +18,7 @@ def openvpn(hello_interval, *args, **kw):
     return subprocess.Popen(args, **kw)
 
 def server(server_ip, ip_length, max_clients, dh_path, pipe_fd, port, proto, hello_interval, *args, **kw):
-    utils.log('Starting server', 3)
+    utils.log('Starting server...', 3)
     return openvpn(hello_interval,
         '--tls-server',
         '--mode', 'server',
@@ -32,7 +32,7 @@ def server(server_ip, ip_length, max_clients, dh_path, pipe_fd, port, proto, hel
         *args, **kw)
 
 def client(server_address, pipe_fd, hello_interval, *args, **kw):
-    utils.log('Starting client', 5)
+    utils.log('Starting client...', 5)
     remote= ['--nobind',
              '--client',
              '--up', 'ovpn-client',
@@ -44,16 +44,16 @@ def client(server_address, pipe_fd, hello_interval, *args, **kw):
 
 def router(network, internal_ip, interface_list,
            wireless, hello_interval, state_path, **kw):
-    utils.log('Starting babel', 3)
+    utils.log('Starting babel...', 3)
     args = ['babeld',
             '-C', 'redistribute local ip %s' % (internal_ip),
             '-C', 'redistribute local deny',
             # Route VIFIB ip adresses
             '-C', 'in ip %s::/%u' % (utils.ipFromBin(network), len(network)),
-            # Route only addresse in the 'local' network,
-            # or other entire networks
-            #'-C', 'in ip %s' % (config.internal_ip),
-            #'-C', 'in ip ::/0 le %s' % network_mask,
+                  # Route only addresse in the 'local' network,
+                  # or other entire networks
+                  #'-C', 'in ip %s' % (config.internal_ip),
+                  #'-C', 'in ip ::/0 le %s' % network_mask,
             # Don't route other addresses
             '-C', 'in deny',
             '-d', str(verbose),

tunnel.py

-import os, random, traceback, time, struct, operator
+import os, random, traceback, time, struct, subprocess, operator
 import plib, utils, db
 
 log = None
@@ -15,7 +15,8 @@ class Connection:
         self.process = plib.client(address, write_pipe, hello, '--dev', iface,
                 *ovpn_args, stdout=os.open(os.path.join(log,
                 'vifibnet.client.%s.log' % (prefix,)),
-                os.O_WRONLY | os.O_CREAT | os.O_TRUNC))
+                os.O_WRONLY | os.O_CREAT | os.O_TRUNC),
+                stderr=subprocess.STDOUT)
 
         self.iface = iface
         self.routes = 0
@@ -66,7 +67,7 @@ class Connection:
 class TunnelManager:
 
     def __init__(self, write_pipe, peer_db, openvpn_args, hello_interval,
-                refresh, connection_count, refresh_rate):
+                refresh, connection_count, refresh_rate, iface_list, network):
         self._write_pipe = write_pipe
         self._peer_db = peer_db
         self._connection_dict = {}
@@ -74,6 +75,10 @@ class TunnelManager:
         self._ovpn_args = openvpn_args
         self._hello = hello_interval
         self._refresh_time = refresh
+        self._network = network
+        self._net_len = len(network)
+        self._iface_list = iface_list
+        self.__indirect_connect = []
         self.free_interface_set = set(('client1', 'client2', 'client3',
                                        'client4', 'client5', 'client6',
                                        'client7', 'client8', 'client9',
@@ -84,11 +89,12 @@ class TunnelManager:
         self._refresh_count = refresh_rate * self._client_count
 
     def refresh(self):
-        utils.log('Refreshing the tunnels', 2)
+        utils.log('Refreshing the tunnels...', 2)
         self._cleanDeads()
         self._countRoutes()
         self._removeSomeTunnels()
         self._makeNewTunnels()
+        utils.log('Tunnels refreshed', 2)
         self.next_refresh = time.time() + self._refresh_time
 
     def _cleanDeads(self):
@@ -107,7 +113,7 @@ class TunnelManager:
             self._kill(prefix)
 
     def _kill(self, prefix):
-        utils.log('Killing the connection with ' + prefix, 2)
+        utils.log('Killing the connection with %s...' % (prefix,), 2)
         connection = self._connection_dict.pop(prefix)
         try:
             connection.process.kill()
@@ -117,9 +123,11 @@ class TunnelManager:
         self.free_interface_set.add(connection.iface)
         self._peer_db.unusePeer(prefix)
         del self._iface_to_prefix[connection.iface]
+        utils.log('Connection with %s killed' % (prefix,), 2)
 
     def _makeNewTunnels(self):
-        utils.log('Trying to make %i new tunnels' %
+        i = 0
+        utils.log('Trying to make %i new tunnels...' %
                 (self._client_count - len(self._connection_dict)), 5)
         try:
             for prefix, address in self._peer_db.getUnusedPeers(
@@ -131,6 +139,8 @@ class TunnelManager:
                         prefix, self._ovpn_args)
                 self._iface_to_prefix[iface] = prefix
                 self._peer_db.usePeer(prefix)
+                i += 1
+            utils.log('%u new tunnels established' % (i,), 3)
         except KeyError:
             utils.log("""Can't establish connection with %s
                     : no available interface""" % prefix, 2)
@@ -138,7 +148,8 @@ class TunnelManager:
             traceback.print_exc()
 
     def _countRoutes(self):
-        utils.log('Starting to count the routes on each interface', 3)
+        utils.log('Starting to count the routes on each interface...', 3)
+        self._indirect_connect = []
         for iface in self._iface_to_prefix.keys():
             self._connection_dict[self._iface_to_prefix[iface]].routes = 0
         f = open('/proc/net/ipv6_route', 'r')
@@ -146,9 +157,21 @@ class TunnelManager:
             ip, subnet_size, iface = struct.unpack('32s x 2s 106x %ss x'
                 % (len(line) - 142), line)
             iface = iface.replace(' ', '')
+            utils.log('Route on iface %s detected to %s/%s'
+                    % (iface, ip, subnet_size), 8)
             if iface in self._iface_to_prefix.keys():
                 self._connection_dict[self._iface_to_prefix[iface]].routes += 1
+            if iface in self._iface_list:
+                subnet_size = int(subnet_size, 16)
+                ip = bin(int(ip, 16))[2:].rjust(128, '0')
+                if self._net_len < subnet_size < 128 and ip.startswith(self._network):
+                    prefix = ip[self._net_len:subnet_size]
+                    utils.log('A route to %s has been discovered on the LAN'
+                            % (prefix,), 3)
+                    self._peer_db.blacklist(prefix)
+        utils.log("Routes have been counted", 3)
         for p in self._connection_dict.keys():
             utils.log('Routes on iface %s : %s' % (
                 self._connection_dict[p].iface,
                 self._connection_dict[p].routes), 5)
+

vifibnet.py

@@ -3,17 +3,18 @@ import argparse, errno, os, select, subprocess, time
 from argparse import ArgumentParser
 import db, plib, upnpigd, utils, tunnel
 
-
 class ArgParser(ArgumentParser):
 
     def convert_arg_line_to_args(self, arg_line):
         arg_line = arg_line.split('#')[0].rstrip()
         if arg_line:
+            if arg_line.startswith('@'):
+                yield arg_line
+                return
             for arg in ('--' + arg_line.lstrip('--')).split():
                 if arg.strip():
                     yield arg
 
-
 def ovpnArgs(optional_args, ca_path, cert_path):
     # Treat openvpn arguments
     if optional_args[0] == "--":
@@ -39,19 +40,17 @@ def getConfig():
             help='Path to vifibnet logs directory')
     _('-s', '--state', default='/var/lib/vifibnet',
             help='Path to VPN state directory')
-    _('--verbose', '-v', default=0, type=int,
+    _('-v', '--verbose', default=0, type=int,
             help='Defines the verbose level')
-    #_('--babel-state', default='/var/lib/vifibnet/babel_state',
-    #        help='Path to babeld state-file')
-    #_('--db', default='/var/lib/vifibnet/peers.db',
-    #        help='Path to peers database')
+    _('-i', '--interface', action='append', dest='iface_list', default=[],
+            help='Extra interface for LAN discovery')
     _('--server', required=True,
             help="VPN address of the discovery peer server")
     _('--server-port', required=True, type=int,
             help="VPN port of the discovery peer server")
 
     # Routing algorithm options
-    _('--hello', type=int, default=30,
+    _('--hello', type=int, default=15,
             help='Hello interval for babel, in seconds')
     _('-w', '--wireless', action='store_true',
             help='''Set all interfaces to be treated as wireless interfaces
@@ -94,17 +93,20 @@ def main():
     tunnel.log = config.log
     utils.verbose = plib.verbose = config.verbose
 
+    utils.log("Configuration :\n" + str(config), 5)
+
     # Create and open read_only pipe to get server events
-    utils.log('Creating pipe for server events', 3)
+    utils.log('Creating pipe for server events...', 3)
     r_pipe, write_pipe = os.pipe()
     read_pipe = os.fdopen(r_pipe)
+    utils.log('Pipe created', 5)
 
     # Init db and tunnels
     forwarder = None
     if manual:
-        utils.log('Manual external configuration', 3)
+        utils.log('Detected manual external configuration', 3)
     else:
-        utils.log('Attempting automatic configuration via UPnP', 4)
+        utils.log('Attempting automatic configuration via UPnP...', 4)
         try:
             forwarder = upnpigd.Forwarder()
             config.address = []
@@ -120,10 +122,11 @@ def main():
             manual, config.pp, 200)
     tunnel_manager = tunnel.TunnelManager(write_pipe, peer_db, openvpn_args,
             config.hello, config.tunnel_refresh, config.connection_count,
-            config.refresh_rate)
+            config.refresh_rate, config.iface_list, network)
 
     # Launch routing protocol. WARNING : you have to be root to start babeld
-    interface_list = ['vifibnet'] + list(tunnel_manager.free_interface_set)
+    interface_list = ['vifibnet'] + list(tunnel_manager.free_interface_set) \
+                     + config.iface_list
     router = plib.router(network, internal_ip, interface_list, config.wireless,
             config.hello, os.path.join(config.state, 'vifibnet.babeld.state'),
             stdout=os.open(os.path.join(config.log, 'vifibnet.babeld.log'),
@@ -135,7 +138,8 @@ def main():
         proto, config.hello, '--dev', 'vifibnet', *openvpn_args,
         stdout=os.open(os.path.join(config.log,
             'vifibnet.server.%s.log' % (proto,)),
-            os.O_WRONLY | os.O_CREAT | os.O_TRUNC))
+            os.O_WRONLY | os.O_CREAT | os.O_TRUNC),
+        stderr=subprocess.STDOUT)
         for port, proto in config.pp)
     tunnel_manager.refresh()