btdht.dht module¶
DHT |
A DHT class ready for instanciation |
DHT_BASE |
The DHT base class |
Node |
A node of the dht in the routing table |
Bucket |
A bucket of nodes in the routing table |
RoutingTable |
A routing table for one or more DHT_BASE instances |
-
class
btdht.dht.
DHT
¶ Bases:
btdht.dht.DHT_BASE
A DHT class ready for instanciation
Parameters: - routing_table (RoutingTable) – An optional routing table, possibly shared between several dht instances. If not specified, a new routing table is instanciated.
- bind_port (int) – And optional udp port to use for the dht instance. If not specified, the hosting system will choose an available port.
- bind_ip (str) – The interface to listen to. The default is
"0.0.0.0"
. - id (bytes) – An optional 160 bits long (20 Bytes) id. If not specified, a random one is generated.
- ignored_ip (set) – A set of ip address in dotted (
"1.2.3.4"
) notation to ignore. The default is the empty set. - debuglvl (int) – Level of verbosity, default to
0
. - prefix (str) – A prefix to use in logged messages. The default is
""
. - process_queue_size (int) – Size of the queue of messages waiting to be processed by user
defines functions (on_`msg`_(query|response)). see the
register_message
method. The default to500
. - ignored_net (list) – An list of ip networks in cidr notation (
"1.2.3.4/5"
) to ignore. The default is the value of the attributeignored_net
. - scheduler (btdht.utils.Scheduler) – A optional
Scheduler
instance. If not specified, a newScheduler
is instanciated.
- Note:
- try to use same
id
andbind_port
over dht restart to increase the probability to remain in other nodes routing table
-
class
btdht.dht.
DHT_BASE
¶ Bases:
object
The DHT base class
Parameters: - routing_table (RoutingTable) – An optional routing table, possibly shared between several dht instances. If not specified, a new routing table is instanciated.
- bind_port (int) – And optional udp port to use for the dht instance. If not specified, the hosting system will choose an available port.
- bind_ip (str) – The interface to listen to. The default is
"0.0.0.0"
. - id (bytes) – An optional 160 bits long (20 Bytes) id. If not specified, a random one is generated.
- ignored_ip (set) – A set of ip address in dotted (
"1.2.3.4"
) notation to ignore. The default is the empty set. - debuglvl (int) – Level of verbosity, default to
0
. - prefix (str) – A prefix to use in logged messages. The default is
""
. - process_queue_size (int) – Size of the queue of messages waiting to be processed by user
defines functions (on_`msg`_(query|response)). see the
register_message
method. The default to500
. - ignored_net (list) – An list of ip networks in cidr notation (
"1.2.3.4/5"
) to ignore. The default is the value of the attributeignored_net
. - scheduler (btdht.utils.Scheduler) – A optional
Scheduler
instance. If not specified, a newScheduler
is instanciated.
- Note:
- try to use same
id
andbind_port
over dht restart to increase the probability to remain in other nodes routing table
-
last_msg
= 0¶ last time we received any message
-
last_msg_rep
= 0¶ last time we receive a response to one of our messages
-
ignored_net
= ['0.0.0.0/8', '10.0.0.0/8', '100.64.0.0/10', '127.0.0.0/8', '169.254.0.0/16', '172.16.0.0/12', '192.0.0.0/24', '192.0.2.0/24', '192.168.0.0/16', '198.18.0.0/15', '198.51.100.0/24', '203.0.113.0/24', '224.0.0.0/4', '240.0.0.0/4', '255.255.255.255/32']¶ list
of default ignored ip networks
-
myid
= None¶ utils.ID
the dht instance id, 160bits long (20 Bytes)
-
root
= None¶ RoutingTable
the used instance of the routing table
-
sock
= None¶ The current dht
socket.socket
-
stoped
= True¶ the state (stoped ?) of the dht
-
token
= defaultdict(<type 'list'>, {})¶ Token send with get_peers response. Map between ip addresses and a list of random token. A new token by ip is genereted at most every 5 min, a single token is valid 10 min. On reception of a announce_peer query from ip, the query is only accepted if we have a valid token (generated less than 10min ago).
-
mytoken
= {}¶ Tokens received on get_peers response. Map between ip addresses and received token from ip. Needed to send announce_peer to that particular ip.
-
transaction_type
= {}¶ Map beetween transaction id and messages type (to be able to match responses)
-
to_send
= <btdht.utils.PollableQueue instance>¶ A
PollableQueue
of messages (data, (ip, port)) to send
-
to_schedule
= []¶ A list of looping iterator to schedule, passed to
_scheduler
-
zombie
¶ True
if dht is stopped but one thread or more remains alive,False
otherwise
-
save
(filename=None, max_node=None)¶ save the current list of nodes to
filename
.Parameters:
-
load
(filename=None, max_node=None)¶ load a list of nodes from
filename
.Parameters:
-
start
(start_routing_table=True, start_scheduler=True)¶ - Start the dht:
- initialize some attributes
- initialize the dht socket (see :meth:init_socket)
- register this instance of the dht in the routing table
(see
RoutingTable.register_dht()
) - register this instance of the dht in the scheduler
- start the routing table if needed and
start_routing_table` is ``True
- start the scheduler if needed and
start_scheduler
isTrue
Parameters:
-
stop
()¶ Stop the dht:
- Set the attribute
stoped
toTrue
and wait for threads to terminate - Close the dht socket
Raises: FailToStop – if there is still some alive threads after 30 secondes, with the list of still alive threads as parameter. - Set the attribute
-
stop_bg
()¶ Lauch the stop process of the dht and return immediately
-
init_socket
()¶ Initialize the UDP socket of the DHT
-
is_alive
()¶ Test if all threads of the dht are alive, stop the dht if one of the thread is dead
Returns: True
if all dht threads are alive,False
otherwise and stop all remaining threads.Return type: bool
-
debug
(lvl, msg)¶ Print
msg
prefixed withprefix
iflvl
<=debuglvl
Parameters: - Note:
- duplicate messages are removed
-
sleep
(t, fstop=None)¶ Sleep for t seconds. If the dht is requested to be stop, run
fstop()
and exitParameters: - t (float) – A time to sleep, in seconds
- fstop – A callable with no arguments, called before exiting
- Note:
- Dont use it in the main thread otherwise it can exit before child threads. Only use it in child threads
) .. automethod:: build_table .. automethod:: announce_peer(info_hash, port, delay=0, block=True) .. automethod:: get_peers(hash, delay=0, block=True, callback=None, limit=10)
-
get_closest_nodes
(id, compact=False)¶ return the current K closest nodes from
id
present in the routing table (K = 8)Parameters: Returns: A list of
Node
ifcompact
isFalse
, abytes
of size multiple of 26 ifcompact
isTrue
.Return type: list
ifcompact
isFalse
, abytes
otherwise.- Note:
Contact information for peers is encoded as a 6-byte string. Also known as “Compact IP-address/port info” the 4-byte IP address is in network byte order with the 2 byte port in network byte order concatenated onto the end.
Contact information for nodes is encoded as a 26-byte string. Also known as “Compact node info” the 20-byte Node ID in network byte order and the compact IP-address/port info concatenated to the end.
-
sendto
(msg, addr)¶ Program a msg to be send over the network
Parameters: - msg (bytes) – The message to send
- addr (tuple) – A couple (ip, port) to send the message to. ip is in dotted notation
- Notes:
- The message is push to the
to_send
queue.
-
clean
()¶ Function called every 15s to do some cleanning. It can safely be overload
-
clean_long
()¶ Function called every 15min to do some cleanning. It can safely be overload
-
register_message
(msg)¶ - Register a dht message to be processed by the following user defined functions
Parameters: msg (bytes) – A dht message to register like b'error'
,b'ping'
,b'find_node'
,b'get_peers'
orb'announce_peer'
- Note:
- on query reception, the function on_``msg``_query will be call with the query as parameter
- on response reception, the function on_``msg``_response will be called with the query and the response as parameters
- on error reception, the function
on_error
will be called with the error and the query as parameter - The message kind is in the
q
key of any dht query message
-
on_announce_peer_response
(query, response)¶ Function called on a announce_peer response reception. Can safely the overloaded
Parameters: - query (krcp.BMessage) – the sent query object
- response (krcp.BMessage) – the received response object
- Notes:
- For this function to be called on announce_peer response reception, you need to call
register_message()
with the parameterb'announce_peer'
-
on_announce_peer_query
(query)¶ Function called on a announce query reception. Can safely the overloaded
Parameters: query (krcp.BMessage) – the received query object - Notes:
- For this function to be called on announce_peer query reception, you need to call
register_message()
with the parameterb'announce_peer'
-
on_find_node_query
(query)¶ Function called on a find_node query reception. Can safely the overloaded
Parameters: query (krcp.BMessage) – the received query object - Notes:
- For this function to be called on find_node query reception, you need to call
register_message()
with the parameterb'find_node'
-
on_find_node_response
(query, response)¶ Function called on a find_node response reception. Can safely the overloaded
Parameters: - query (krcp.BMessage) – the sent query object
- response (krcp.BMessage) – the received response object
- Notes:
- For this function to be called on find_node response reception, you need to call
register_message()
with the parameterb'find_node'
-
on_get_peers_query
(query)¶ Function called on a get_peers query reception. Can safely the overloaded
Parameters: query (krcp.BMessage) – the received query object - Notes:
- For this function to be called on get_peers query reception, you need to call
register_message()
with the parameterb'get_peers'
-
on_get_peers_response
(query, response)¶ Function called on a get_peers response reception. Can safely the overloaded
Parameters: - query (krcp.BMessage) – the sent query object
- response (krcp.BMessage) – the received response object
- Notes:
- For this function to be called on get_peers response reception, you need to call
register_message()
with the parameterb'get_peers'
-
on_ping_query
(query)¶ Function called on a ping query reception. Can safely the overloaded
Parameters: query (krcp.BMessage) – the received query object - Notes:
- For this function to be called on ping query reception, you need to call
register_message()
with the parameterb'ping'
-
on_ping_response
(query, response)¶ Function called on a ping response reception. Can safely the overloaded
Parameters: - query (krcp.BMessage) – the sent query object
- response (krcp.BMessage) – the received response object
- Notes:
- For this function to be called on ping response reception, you need to call
register_message()
with the parameterb'ping'
-
on_error
(error, query=None)¶ Function called then a query has be responded by an error message. Can safely the overloaded.
Parameters: - error (krcp.Berror) – An error instance
- query (krcp.BMessage) – An optional query raising the error message
- Notes:
- For this function to be called on error reception, you need to call
register_message()
with the parameterb'error'
-
class
btdht.dht.
Node
¶ Bases:
object
A node of the dht in the routing table
Parameters: - id (bytes) – The 160 bits (20 Bytes) long identifier of the node
- ip (str) – The ip, in dotted notation of the node
- port (int) – The udp dht port of the node
- last_response (int) – Unix timestamp of the last received response from this node
- last_query (int) – Unix timestamp of the last received query from this node
- failed (int) – Number of consecutive queries sended to the node without responses
- Note:
- A good node is a node has responded to one of our queries within the last 15 minutes. A node is also good if it has ever responded to one of our queries and has sent us a query within the last 15 minutes. After 15 minutes of inactivity, a node becomes questionable. Nodes become bad when they fail to respond to multiple queries in a row (3 query in a row in this implementation).
-
port
¶ UDP port of the node
-
last_response
¶ Unix timestamp of the last received response from this node
-
last_query
¶ Unix timestamp of the last received query from this node
-
failed
¶ Number of reponse pending (increase on sending query to the node, set to 0 on reception from the node)
-
id
¶ 160bits (20 Bytes) identifier of the node
-
good
¶ True
if the node is a good node. A good node is a node has responded to one of our queries within the last 15 minutes. A node is also good if it has ever responded to one of our queries and has sent us a query within the last 15 minutes.
-
bad
¶ True
if the node is a bad node (communication with the node is not possible). Nodes become bad when they fail to respond to 3 queries in a row.
-
ip
¶ IP address of the node in dotted notation
-
compact_info
()¶ Return the compact contact information of the node
- Notes:
- Contact information for peers is encoded as a 6-byte string. Also known as “Compact IP-address/port info” the 4-byte IP address is in network byte order with the 2 byte port in network byte order concatenated onto the end. Contact information for nodes is encoded as a 26-byte string. Also known as “Compact node info” the 20-byte Node ID in network byte order has the compact IP-address/port info concatenated to the end.
-
from_compact_infos
(infos)¶ This is a classmethod
Instancy nodes from multiple compact node information string
Parameters: infos (bytes) – A string of size multiple of 26 Returns: A list of Node
instancesReturn type: list - Notes:
- Contact information for peers is encoded as a 6-byte string. Also known as “Compact IP-address/port info” the 4-byte IP address is in network byte order with the 2 byte port in network byte order concatenated onto the end. Contact information for nodes is encoded as a 26-byte string. Also known as “Compact node info” the 20-byte Node ID in network byte order has the compact IP-address/port info concatenated to the end.
-
from_compact_info
(info)¶ This is a classmethod
Instancy a node from its compact node infoformation string
Parameters: info (bytes) – A string of length 26 Returns: A node instance Return type: Node - Notes:
- Contact information for peers is encoded as a 6-byte string. Also known as “Compact IP-address/port info” the 4-byte IP address is in network byte order with the 2 byte port in network byte order concatenated onto the end. Contact information for nodes is encoded as a 26-byte string. Also known as “Compact node info” the 20-byte Node ID in network byte order has the compact IP-address/port info concatenated to the end.
-
announce_peer
(dht, info_hash, port)¶ Send a announce_peer query to the node
Parameters: Raises: NoTokenError – if we have no valid token for
info_hash
. Try to callget_peers()
on thisinfo_hash
first.
-
find_node
(dht, target)¶ Send a find_node query to the node
Parameters: - dht (DHT_BASE) – The dht instance to use to send the message
- target (bytes) – the 160bits (20 bytes) target node id
-
class
btdht.dht.
Bucket
¶ Bases:
list
A bucket of nodes in the routing table
Parameters: - id (bytes) – A prefix identifier from 0 to 169 bits for the bucket
- id_length (int) – number of signifiant bit in
id
(can also be seen as the length between the root and the bucket in the routing table) - init (iterable) – some values to store initialy in the bucket
-
max_size
= 8¶ Maximun number of element in the bucket
-
last_changed
= 0¶ Unix timestamp, last time the bucket had been updated
-
id
= None¶ A prefix identifier from 0 to 160 bits for the bucket
-
to_refresh
¶ True
if the bucket need refreshing
-
random_id
()¶ Returns: A random id handle by the bucket Return type: bytes This is used to send find_nodes for randoms ids in a bucket
-
add
(dht, node)¶ Try to add a node to the bucket.
Parameters: Raises: BucketFull – if the bucket is full
- Notes:
The addition of a node to a bucket is done as follow: * if the bucket is not full, just add the node * if the bucket is full
- if there is some bad nodes in the bucket, remove a bad node and add the node
- if there is some questionnable nodes (neither good not bad), send a ping request to the oldest one, discard the node
- if all nodes are good in the bucket, discard the node
-
get_node
(id)¶ Returns: A Node
withNode.id
equal toid
Return type: Node Raises: NotFound – if no node is found within this bucket
-
own
(id)¶ Parameters: id (bytes) – A 60bit (20 Bytes) identifier Returns: True
ifid
is handled by this bucketReturn type: bool
-
split
(rt, dht)¶ Split the bucket into two buckets
Parameters: - rt (RoutingTable) – The routing table handling the bucket
- dht (DHT_BASE) – A dht using
rt
as routing table
Returns: A couple of two bucket, the first one this the last significant bit of its id equal to 0, the second, equal to 1
Return type: Raises: BucketNotFull – If the bucket has not
max_size
elements (and so the split is not needed)
-
class
btdht.dht.
RoutingTable
¶ Bases:
object
A routing table for one or more
DHT_BASE
instancesParameters: - scheduler (utils.Scheduler) – A scheduler instance
- debuglvl (int) – Level of verbosity, default to
0
.
-
trie
= None¶ The routing table storage data structure, an instance of
datrie.Trie
-
stoped
= True¶ The state (stoped ?) of the routing table
-
need_merge
= False¶ Is a merge sheduled ?
-
to_schedule
= []¶ A class:list of couple (weightless thread name, weightless thread function)
-
prefix
= ''¶ Prefix in logs and threads name
-
zombie
¶ True
if dht is stopped but one thread or more remains alive,False
otherwise
-
start
()¶ start the routing table
-
stop
()¶ stop the routing table and wait for all threads to terminate
-
stop_bg
()¶ stop the routing table and return immediately
-
is_alive
()¶ Test if all routing table threads are alive. If a thread is found dead, stop the routingtable
Returns: True
if all routing table threads are alive,False
otherwiseReturn type: bool
-
register_torrent
(id)¶ Register a torrent
id
(info_hash) for being tracked by the routing table. This means that if a node need to be added to the bucket handling ``id``and the bucket is full, then, this bucket will be split into 2 bucketsParameters: id (bytes) – A 160 bits (20 Bytes) torrent identifier - Note:
- torrent ids can automaticaly be release by a dht instance after a get_peers.
For keeping a torrent registered, use the method
register_torrent_longterm()
-
release_torrent
(id)¶ Release a torrent
id
(info_hash) and program the routing table to be mergedParameters: id (bytes) – A 160 bits (20 Bytes) torrent identifier
-
register_torrent_longterm
(id)¶ Same as
register_torrent()
but garanty that the torrent wont be released automaticaly by the dht.Parameters: id (bytes) – A 160 bits (20 Bytes) torrent identifier
-
release_torrent_longterm
(id)¶ For releasing torrent registered with the :meth`register_torrent_longterm` method
Parameters: id (bytes) – A 160 bits (20 Bytes) torrent identifier
-
register_dht
(dht)¶ Register a
dht
instance to the routing tableParameters: dht (DHT_BASE) – A dht instance - Notes:
- on start, all dht instances automaticaly register themself to their routing tables
-
release_dht
(dht)¶ Release a
dht
instance to the routing table, and shedule the routing table for a merge.- Notes:
- on stop, dht automatially release itself from the routing table
-
empty
()¶ Empty the routing table, deleting all buckets
-
debug
(lvl, msg)¶ same as
DHT_BASE.debug()
-
stats
()¶ Returns: A triple (number of nodes, number of good nodes, number of bad nodes) Return type: tuple
-
find
(id)¶ Parameters: id (bytes) – A 160 bits (20 Bytes) identifier Returns: The bucket handling id
Return type: Bucket Raises: KeyError – then a racing condition with merging and/or spliting a bucket is met. This should not happen - Notes:
- Duging a split or merge of bucket it is possible that the bucket handling
id
is not found.find()
will retry at most 20 times to get the bucket. In most case, during those retries, the split and/or merge will end and the bucket handlingid
will be returned.
-
get_node
(id)¶ Parameters: id (bytes) – A 160 bits (20 Bytes) identifier Returns: A node with id id
Return type: Node Raises: NotFound – if no nodes is found
-
get_closest_nodes
(id, bad=False)¶ Return the K closest nodes from
id
in the routing tableParameters: - id (bytes) – A 160 bits (20 Bytes) identifier
- bad (bool) – Should we return bad nodes ? The default is
False
- Notes:
- If less than K (=8) good nodes is found, bad nodes will be included it solve the case there the connection where temporary lost and all nodes in the routing table marked as bad. In normal operation, we should always find K (=8) good nodes in the routing table.
-
add
(dht, node)¶ Add a node the the routing table
Parameters:
-
split
(dht, bucket)¶ Split
bucket
in twoParameters: - Notes:
- the routing table cover the entire 160bits space
-
merge
()¶ Request a merge to be perform