btdht.dht module

DHT A DHT class ready for instanciation
DHT_BASE The DHT base class
Node A node of the dht in the routing table
Bucket A bucket of nodes in the routing table
RoutingTable A routing table for one or more DHT_BASE instances
class btdht.dht.DHT

Bases: btdht.dht.DHT_BASE

A DHT class ready for instanciation

Parameters:
  • routing_table (RoutingTable) – An optional routing table, possibly shared between several dht instances. If not specified, a new routing table is instanciated.
  • bind_port (int) – And optional udp port to use for the dht instance. If not specified, the hosting system will choose an available port.
  • bind_ip (str) – The interface to listen to. The default is "0.0.0.0".
  • id (bytes) – An optional 160 bits long (20 Bytes) id. If not specified, a random one is generated.
  • ignored_ip (set) – A set of ip address in dotted ("1.2.3.4") notation to ignore. The default is the empty set.
  • debuglvl (int) – Level of verbosity, default to 0.
  • prefix (str) – A prefix to use in logged messages. The default is "".
  • process_queue_size (int) – Size of the queue of messages waiting to be processed by user defines functions (on_`msg`_(query|response)). see the register_message method. The default to 500.
  • ignored_net (list) – An list of ip networks in cidr notation ("1.2.3.4/5") to ignore. The default is the value of the attribute ignored_net.
  • scheduler (btdht.utils.Scheduler) – A optional Scheduler instance. If not specified, a new Scheduler is instanciated.
Note:
try to use same id and bind_port over dht restart to increase the probability to remain in other nodes routing table
class btdht.dht.DHT_BASE

Bases: object

The DHT base class

Parameters:
  • routing_table (RoutingTable) – An optional routing table, possibly shared between several dht instances. If not specified, a new routing table is instanciated.
  • bind_port (int) – And optional udp port to use for the dht instance. If not specified, the hosting system will choose an available port.
  • bind_ip (str) – The interface to listen to. The default is "0.0.0.0".
  • id (bytes) – An optional 160 bits long (20 Bytes) id. If not specified, a random one is generated.
  • ignored_ip (set) – A set of ip address in dotted ("1.2.3.4") notation to ignore. The default is the empty set.
  • debuglvl (int) – Level of verbosity, default to 0.
  • prefix (str) – A prefix to use in logged messages. The default is "".
  • process_queue_size (int) – Size of the queue of messages waiting to be processed by user defines functions (on_`msg`_(query|response)). see the register_message method. The default to 500.
  • ignored_net (list) – An list of ip networks in cidr notation ("1.2.3.4/5") to ignore. The default is the value of the attribute ignored_net.
  • scheduler (btdht.utils.Scheduler) – A optional Scheduler instance. If not specified, a new Scheduler is instanciated.
Note:
try to use same id and bind_port over dht restart to increase the probability to remain in other nodes routing table
bind_ip = '0.0.0.0'

str interface the dht is binded to

bind_port = None

int port the dht is binded to

debuglvl = 0

int the dht instance verbosity level

last_msg = 0

last time we received any message

last_msg_rep = 0

last time we receive a response to one of our messages

ignored_ip = []

set of ignored ip in dotted notation

ignored_net = ['0.0.0.0/8', '10.0.0.0/8', '100.64.0.0/10', '127.0.0.0/8', '169.254.0.0/16', '172.16.0.0/12', '192.0.0.0/24', '192.0.2.0/24', '192.168.0.0/16', '198.18.0.0/15', '198.51.100.0/24', '203.0.113.0/24', '224.0.0.0/4', '240.0.0.0/4', '255.255.255.255/32']

list of default ignored ip networks

myid = None

utils.ID the dht instance id, 160bits long (20 Bytes)

prefix = ''

str prefixing all debug message

root = None

RoutingTable the used instance of the routing table

sock = None

The current dht socket.socket

stoped = True

the state (stoped ?) of the dht

threads = []

list of the Thread of the dht instance

token = defaultdict(<type 'list'>, {})

Token send with get_peers response. Map between ip addresses and a list of random token. A new token by ip is genereted at most every 5 min, a single token is valid 10 min. On reception of a announce_peer query from ip, the query is only accepted if we have a valid token (generated less than 10min ago).

mytoken = {}

Tokens received on get_peers response. Map between ip addresses and received token from ip. Needed to send announce_peer to that particular ip.

transaction_type = {}

Map beetween transaction id and messages type (to be able to match responses)

to_send = <btdht.utils.PollableQueue instance>

A PollableQueue of messages (data, (ip, port)) to send

to_schedule = []

A list of looping iterator to schedule, passed to _scheduler

zombie

True if dht is stopped but one thread or more remains alive, False otherwise

save(filename=None, max_node=None)

save the current list of nodes to filename.

Parameters:
  • filename (str) – An optional filename where to save the current list of nodes. If not provided, the file "dht_`myid`.status is used.
  • max_node (int) – An optional integer to limit the number of saved nodes. If not provided, all of the routing table nodes are saved.
load(filename=None, max_node=None)

load a list of nodes from filename.

Parameters:
  • filename (str) – An optional filename where to load the list of nodes. If not provided, the file "dht_`myid`.status is used.
  • max_node (int) – An optional integer to limit the number of loaded nodes. If not provided, all of the file nodes are loaded.
start(start_routing_table=True, start_scheduler=True)
Start the dht:
  • initialize some attributes
  • initialize the dht socket (see :meth:init_socket)
  • register this instance of the dht in the routing table (see RoutingTable.register_dht())
  • register this instance of the dht in the scheduler
  • start the routing table if needed and start_routing_table` is ``True
  • start the scheduler if needed and start_scheduler is True
Parameters:
  • start_routing_table (bool) – If True (the default) also start the routing table if needed
  • start_scheduler (bool) – If ``True``(the default) alsp start the scheduler
stop()

Stop the dht:

  • Set the attribute stoped to True and wait for threads to terminate
  • Close the dht socket
Raises:FailToStop – if there is still some alive threads after 30 secondes, with the list of still alive threads as parameter.
stop_bg()

Lauch the stop process of the dht and return immediately

init_socket()

Initialize the UDP socket of the DHT

is_alive()

Test if all threads of the dht are alive, stop the dht if one of the thread is dead

Returns:True if all dht threads are alive, False otherwise and stop all remaining threads.
Return type:bool
debug(lvl, msg)

Print msg prefixed with prefix if lvl <= debuglvl

Parameters:
  • lvl (int) – The debug level of the message to print
  • msg (str) – The debug message to print
Note:
duplicate messages are removed
sleep(t, fstop=None)

Sleep for t seconds. If the dht is requested to be stop, run fstop() and exit

Parameters:
  • t (float) – A time to sleep, in seconds
  • fstop – A callable with no arguments, called before exiting
Note:
Dont use it in the main thread otherwise it can exit before child threads. Only use it in child threads

) .. automethod:: build_table .. automethod:: announce_peer(info_hash, port, delay=0, block=True) .. automethod:: get_peers(hash, delay=0, block=True, callback=None, limit=10)

get_closest_nodes(id, compact=False)

return the current K closest nodes from id present in the routing table (K = 8)

Parameters:
  • id (bytes) – A 160bits (20 Bytes) long identifier for which we want the closest nodes in the routing table.
  • compact (bool) – If True the nodes infos are returned in compact format. Otherwise, intances of Node are returned. The default is False.
Returns:

A list of Node if compact is False, a bytes of size multiple of 26 if compact is True.

Return type:

list if compact is False, a bytes otherwise.

Note:

Contact information for peers is encoded as a 6-byte string. Also known as “Compact IP-address/port info” the 4-byte IP address is in network byte order with the 2 byte port in network byte order concatenated onto the end.

Contact information for nodes is encoded as a 26-byte string. Also known as “Compact node info” the 20-byte Node ID in network byte order and the compact IP-address/port info concatenated to the end.

sendto(msg, addr)

Program a msg to be send over the network

Parameters:
  • msg (bytes) – The message to send
  • addr (tuple) – A couple (ip, port) to send the message to. ip is in dotted notation
Notes:
The message is push to the to_send queue.
clean()

Function called every 15s to do some cleanning. It can safely be overload

clean_long()

Function called every 15min to do some cleanning. It can safely be overload

register_message(msg)
Register a dht message to be processed by the following user defined functions
Parameters:msg (bytes) – A dht message to register like b'error', b'ping', b'find_node', b'get_peers' or b'announce_peer'
Note:
  • on query reception, the function on_``msg``_query will be call with the query as parameter
  • on response reception, the function on_``msg``_response will be called with the query and the response as parameters
  • on error reception, the function on_error will be called with the error and the query as parameter
  • The message kind is in the q key of any dht query message
on_announce_peer_response(query, response)

Function called on a announce_peer response reception. Can safely the overloaded

Parameters:
Notes:
For this function to be called on announce_peer response reception, you need to call register_message() with the parameter b'announce_peer'
on_announce_peer_query(query)

Function called on a announce query reception. Can safely the overloaded

Parameters:query (krcp.BMessage) – the received query object
Notes:
For this function to be called on announce_peer query reception, you need to call register_message() with the parameter b'announce_peer'
on_find_node_query(query)

Function called on a find_node query reception. Can safely the overloaded

Parameters:query (krcp.BMessage) – the received query object
Notes:
For this function to be called on find_node query reception, you need to call register_message() with the parameter b'find_node'
on_find_node_response(query, response)

Function called on a find_node response reception. Can safely the overloaded

Parameters:
Notes:
For this function to be called on find_node response reception, you need to call register_message() with the parameter b'find_node'
on_get_peers_query(query)

Function called on a get_peers query reception. Can safely the overloaded

Parameters:query (krcp.BMessage) – the received query object
Notes:
For this function to be called on get_peers query reception, you need to call register_message() with the parameter b'get_peers'
on_get_peers_response(query, response)

Function called on a get_peers response reception. Can safely the overloaded

Parameters:
Notes:
For this function to be called on get_peers response reception, you need to call register_message() with the parameter b'get_peers'
on_ping_query(query)

Function called on a ping query reception. Can safely the overloaded

Parameters:query (krcp.BMessage) – the received query object
Notes:
For this function to be called on ping query reception, you need to call register_message() with the parameter b'ping'
on_ping_response(query, response)

Function called on a ping response reception. Can safely the overloaded

Parameters:
Notes:
For this function to be called on ping response reception, you need to call register_message() with the parameter b'ping'
on_error(error, query=None)

Function called then a query has be responded by an error message. Can safely the overloaded.

Parameters:
  • error (krcp.Berror) – An error instance
  • query (krcp.BMessage) – An optional query raising the error message
Notes:
For this function to be called on error reception, you need to call register_message() with the parameter b'error'
class btdht.dht.Node

Bases: object

A node of the dht in the routing table

Parameters:
  • id (bytes) – The 160 bits (20 Bytes) long identifier of the node
  • ip (str) – The ip, in dotted notation of the node
  • port (int) – The udp dht port of the node
  • last_response (int) – Unix timestamp of the last received response from this node
  • last_query (int) – Unix timestamp of the last received query from this node
  • failed (int) – Number of consecutive queries sended to the node without responses
Note:
A good node is a node has responded to one of our queries within the last 15 minutes. A node is also good if it has ever responded to one of our queries and has sent us a query within the last 15 minutes. After 15 minutes of inactivity, a node becomes questionable. Nodes become bad when they fail to respond to multiple queries in a row (3 query in a row in this implementation).
port

UDP port of the node

last_response

Unix timestamp of the last received response from this node

last_query

Unix timestamp of the last received query from this node

failed

Number of reponse pending (increase on sending query to the node, set to 0 on reception from the node)

id

160bits (20 Bytes) identifier of the node

good

True if the node is a good node. A good node is a node has responded to one of our queries within the last 15 minutes. A node is also good if it has ever responded to one of our queries and has sent us a query within the last 15 minutes.

bad

True if the node is a bad node (communication with the node is not possible). Nodes become bad when they fail to respond to 3 queries in a row.

ip

IP address of the node in dotted notation

compact_info()

Return the compact contact information of the node

Notes:
Contact information for peers is encoded as a 6-byte string. Also known as “Compact IP-address/port info” the 4-byte IP address is in network byte order with the 2 byte port in network byte order concatenated onto the end. Contact information for nodes is encoded as a 26-byte string. Also known as “Compact node info” the 20-byte Node ID in network byte order has the compact IP-address/port info concatenated to the end.
from_compact_infos(infos)

This is a classmethod

Instancy nodes from multiple compact node information string

Parameters:infos (bytes) – A string of size multiple of 26
Returns:A list of Node instances
Return type:list
Notes:
Contact information for peers is encoded as a 6-byte string. Also known as “Compact IP-address/port info” the 4-byte IP address is in network byte order with the 2 byte port in network byte order concatenated onto the end. Contact information for nodes is encoded as a 26-byte string. Also known as “Compact node info” the 20-byte Node ID in network byte order has the compact IP-address/port info concatenated to the end.
from_compact_info(info)

This is a classmethod

Instancy a node from its compact node infoformation string

Parameters:info (bytes) – A string of length 26
Returns:A node instance
Return type:Node
Notes:
Contact information for peers is encoded as a 6-byte string. Also known as “Compact IP-address/port info” the 4-byte IP address is in network byte order with the 2 byte port in network byte order concatenated onto the end. Contact information for nodes is encoded as a 26-byte string. Also known as “Compact node info” the 20-byte Node ID in network byte order has the compact IP-address/port info concatenated to the end.
announce_peer(dht, info_hash, port)

Send a announce_peer query to the node

Parameters:
  • dht (DHT_BASE) – The dht instance to use to send the message
  • info_hash (bytes) – A 160bits (20 bytes) torrent id to announce
  • port (int) – The tcp port where data for info_hash is available
Raises:

NoTokenError – if we have no valid token for info_hash. Try to call get_peers() on this info_hash first.

find_node(dht, target)

Send a find_node query to the node

Parameters:
  • dht (DHT_BASE) – The dht instance to use to send the message
  • target (bytes) – the 160bits (20 bytes) target node id
get_peers(dht, info_hash)

Send a get_peers query to the node

Parameters:
  • dht (DHT_BASE) – The dht instance to use to send the message
  • info_hash (bytes) – a 160bits (20 bytes) torrent id
ping(dht)

Send a ping query to the node

Parameters:dht (DHT_BASE) – The dht instance to use to send the message
class btdht.dht.Bucket

Bases: list

A bucket of nodes in the routing table

Parameters:
  • id (bytes) – A prefix identifier from 0 to 169 bits for the bucket
  • id_length (int) – number of signifiant bit in id (can also be seen as the length between the root and the bucket in the routing table)
  • init (iterable) – some values to store initialy in the bucket
max_size = 8

Maximun number of element in the bucket

last_changed = 0

Unix timestamp, last time the bucket had been updated

id = None

A prefix identifier from 0 to 160 bits for the bucket

id_length = 0

Number of signifiant bit in id

to_refresh

True if the bucket need refreshing

random_id()
Returns:A random id handle by the bucket
Return type:bytes

This is used to send find_nodes for randoms ids in a bucket

add(dht, node)

Try to add a node to the bucket.

Parameters:
  • dht (DHT_BASE) – The dht instance the node to add is from
  • node (Node) – A node to add to the bucket
Raises:

BucketFull – if the bucket is full

Notes:

The addition of a node to a bucket is done as follow: * if the bucket is not full, just add the node * if the bucket is full

  • if there is some bad nodes in the bucket, remove a bad node and add the node
  • if there is some questionnable nodes (neither good not bad), send a ping request to the oldest one, discard the node
  • if all nodes are good in the bucket, discard the node
get_node(id)
Returns:A Node with Node.id equal to id
Return type:Node
Raises:NotFound – if no node is found within this bucket
own(id)
Parameters:id (bytes) – A 60bit (20 Bytes) identifier
Returns:True if id is handled by this bucket
Return type:bool
split(rt, dht)

Split the bucket into two buckets

Parameters:
  • rt (RoutingTable) – The routing table handling the bucket
  • dht (DHT_BASE) – A dht using rt as routing table
Returns:

A couple of two bucket, the first one this the last significant bit of its id equal to 0, the second, equal to 1

Return type:

tuple

Raises:

BucketNotFull – If the bucket has not max_size elements (and so the split is not needed)

merge(bucket)

Merge the bucket with bucket

Parameters:bucket (Bucket) – a bucket to be merged with
Returns:The merged bucket
Return type:Bucket
class btdht.dht.RoutingTable

Bases: object

A routing table for one or more DHT_BASE instances

Parameters:
  • scheduler (utils.Scheduler) – A scheduler instance
  • debuglvl (int) – Level of verbosity, default to 0.
debuglvl = 0

int the routing table instance verbosity level

trie = None

The routing table storage data structure, an instance of datrie.Trie

stoped = True

The state (stoped ?) of the routing table

need_merge = False

Is a merge sheduled ?

threads = []

list of the Thread of the routing table instance

to_schedule = []

A class:list of couple (weightless thread name, weightless thread function)

prefix = ''

Prefix in logs and threads name

zombie

True if dht is stopped but one thread or more remains alive, False otherwise

start()

start the routing table

stop()

stop the routing table and wait for all threads to terminate

stop_bg()

stop the routing table and return immediately

is_alive()

Test if all routing table threads are alive. If a thread is found dead, stop the routingtable

Returns:True if all routing table threads are alive, False otherwise
Return type:bool
register_torrent(id)

Register a torrent id (info_hash) for being tracked by the routing table. This means that if a node need to be added to the bucket handling ``id``and the bucket is full, then, this bucket will be split into 2 buckets

Parameters:id (bytes) – A 160 bits (20 Bytes) torrent identifier
Note:
torrent ids can automaticaly be release by a dht instance after a get_peers. For keeping a torrent registered, use the method register_torrent_longterm()
release_torrent(id)

Release a torrent id (info_hash) and program the routing table to be merged

Parameters:id (bytes) – A 160 bits (20 Bytes) torrent identifier
register_torrent_longterm(id)

Same as register_torrent() but garanty that the torrent wont be released automaticaly by the dht.

Parameters:id (bytes) – A 160 bits (20 Bytes) torrent identifier
release_torrent_longterm(id)

For releasing torrent registered with the :meth`register_torrent_longterm` method

Parameters:id (bytes) – A 160 bits (20 Bytes) torrent identifier
register_dht(dht)

Register a dht instance to the routing table

Parameters:dht (DHT_BASE) – A dht instance
Notes:
on start, all dht instances automaticaly register themself to their routing tables
release_dht(dht)

Release a dht instance to the routing table, and shedule the routing table for a merge.

Notes:
on stop, dht automatially release itself from the routing table
empty()

Empty the routing table, deleting all buckets

debug(lvl, msg)

same as DHT_BASE.debug()

stats()
Returns:A triple (number of nodes, number of good nodes, number of bad nodes)
Return type:tuple
heigth()
Returns:the height of the tree of the routing table
Return type:int
find(id)
Parameters:id (bytes) – A 160 bits (20 Bytes) identifier
Returns:The bucket handling id
Return type:Bucket
Raises:KeyError – then a racing condition with merging and/or spliting a bucket is met. This should not happen
Notes:
Duging a split or merge of bucket it is possible that the bucket handling id is not found. find() will retry at most 20 times to get the bucket. In most case, during those retries, the split and/or merge will end and the bucket handling id will be returned.
get_node(id)
Parameters:id (bytes) – A 160 bits (20 Bytes) identifier
Returns:A node with id id
Return type:Node
Raises:NotFound – if no nodes is found
get_closest_nodes(id, bad=False)

Return the K closest nodes from id in the routing table

Parameters:
  • id (bytes) – A 160 bits (20 Bytes) identifier
  • bad (bool) – Should we return bad nodes ? The default is False
Notes:
If less than K (=8) good nodes is found, bad nodes will be included it solve the case there the connection where temporary lost and all nodes in the routing table marked as bad. In normal operation, we should always find K (=8) good nodes in the routing table.
add(dht, node)

Add a node the the routing table

Parameters:
  • dht (DHT_BASE) – The dht instance ``node``is from
  • node (Node) – The node to add to the routing table
split(dht, bucket)

Split bucket in two

Parameters:
  • dht (DHT_BASE) – A dht instance
  • bucket (Bucket) – A bucket from the routing table to split
Notes:
the routing table cover the entire 160bits space
merge()

Request a merge to be perform