Transport Plugins: Update (GSoC 2012)
Hello,
This is an update on the first stage of my project. I have been
working out the details regarding modifications in the core to
simplify the implementation of the transport plugins. This refactoring
stage will involve some minor modifications and some major ones. These
are based on the discussions on #freenet and #freenet-chat. However I
would like to avoid doing changes that would bring about major
differences in the way things function at present.
I have planned to finish refactoring by May 21 (official coding begins
for GSoC). I intend to do that in 3 stages. I have also simultaneously
discussed the entire implementation of transport plugins here. I will
review the feedback and this report, to evaluate what constitutes the
refactoring. After those changes are made I can work on implementing
the needed core level changes, which mostly involve design for
streams.
1) Cryptography and Node
This will involve mainly the classes NodeCrypto and Node. I intend to
make NodeCrypto function purely for keys and not do any network
related task. Presently it also handles binding to the port, selection
of ports for Darknet mode and Opennet mode.
I am considering moving these portions to TransportManager and
PortManager classes.
TransportManager - A registry of registered transports with other
information. Initially I ll dump the refactor changes of UDP into it.
Later on, once UDP is ready as a plugin, along with TCP, it ll be a
generic class for any transport. There will be two objects of this -
one each for opennet and darknet modes. NodeCrypto e.g. queries
TransportManager to find all the transports' addresses (as strings) to
put in the noderef.
Now UDPSocketHandler is equivalent to a UDP transport plugin, only
it's currently part of the core of freenet. So for the moment
TransportManager can communicate with UDPSocketHandler. That I believe
will be an intermediate step before actually converting
UDPSokcetHandler to a plugin.
PortManager - A registry of all active ports used by freenet - 1) List
of ports on fproxy page 2) List of ports for UPnP
Must contain - 1)Port number 2)Socket(TCP/UDP) 3)Forwarded
4)Description 5)Transport reference
However PortManager is second priority, and will not be addressed if
time does not permit.
Also a major refactoring change would be regarding the fact that
objects no longer take a socket or a port for granted. All queries
must go through the TransportManager. I would need help at certain
places to figure out how this can achieved. For e.g.
NodeCrypto/Node/some other classes assume that an unique darknet port
exists. toad suggested we use a debugging identifier here(some default
number, etc). That makes it simpler for me too.
2) Messages and Packets
These part have been discussed on #freenet(and #freenet-chat) and here
are some issues and conclusions-
a. The existing architecture supports only packet based transports.
The first step would be to draw a clear distinction between
packet-based(connectionless sockets) and stream-based(connection-based
sockets). Thus we will need separate classes for streams(with a base
class for streams and packets), that will handle Message-in-flight and
Packets-in-flight, tracking them, allowing for transports to be
switched midway. This will be handled differently in streams. For
Packets the existing classes can be used, however with modifications
to address the following.
b. For streams we need two features - To handle messages as a whole
entity, and acknowledge once an entire message has been received; and
To support partial messages, if for e.g. we have just switched from
UDP to TCP or we are sending a really large message on TCP.
c. NewPacketFormat currently 1) takes a byte[] from UdpSocketHandler
and decodes it 2) creates and sends a packet. Now the idea is to
create NewStreamFormat that handles streams, and also provide a pair
of streams StreamConnectionHandler for the plugin to write/read.
NewPacketFormat handles the list of message-in-flight. A new object
must be used for this purpose while NewPacketFormat performs lesser
tasks. Both NewPacketFormat and NewStreamFormat can communicate with
it.
Question: MessageWrapper tracks a message-in-flight. But is it really
necessary to create a base class for MessageWrapper and create
StreamMessageWrapper too. Or would it be simple to make necessary
modifications(if there are any) to MessageWrapper.
Please explain the purpose of doing so, as I have very little
understanding, as of now, of the object Message. Also how it will be
treated differently for streams against packets.
d. Another issue is regarding FNPPacketMangler. This is an important
aspect of freenet, as it deals with the handshake between two nodes,
and the exchange of cryptographic keys. So the issue being whether per
NodeCrypto we have an object or per transport we have an object, of
the FNPPacketMangler type. And assuming these are packet based
transports.
Question: How can the equivalent of FNPPacketMangler be created for
stream based connections. I don't think this was discussed at all or
in detail on the IRC. Some suggestions here would be helpful. My
knowledge is that the JFK protocol works for packet based transports,
specifically UDP. How would this function in case of streams?
e. Another thing that needs clarification is regarding who decides
when to send the packets. Since stenography is all about the timing
and the format of the packet, the plugin must have control. However
that would make some other transport plugins difficult to write. toad
suggested the following. To quote him- "PacketSender constantly loops
over peers to send packets to them. We'll probably have two separate
mechanisms for packet-based - one where we are just wrapping packets
with different headers, so the node decides when to send a packet, and
one where the plugin decides when to send a packet and demands data to
fill it from the node. The latter is "constant bitrate", or really
accurate fakery; it's a long term project, as is how to fill up the
packets when there's nothing to send." This sounds acceptable, however
when I get there I will discuss how this should be implemented. But I
believe that it will not affect the initial stages of the project.
Question: How does the same work for streams? Firstly the transport
must have its own flow control mechanism to decide when to send. Also
there must be a way to let the node know this. I am guessing
NewStreamFormat handle this? It must also create
StreamConnectionHandler for the plugin to read/write. Must also do
encryption using existing BlockCiphers? It must convert the encrypted
data into a stream? I suppose I can draw parallels to NewPacketFormat
and do it.
3) Plugins, streams, transports
I will need to create a base class for transports - TransportPlugin.
This is designed as per the developer's requirements. However the
major idea is that it would open sockets, and listen for connections.
For a packet-based plugin as mentioned it is called by PacketSender to
send data. If it wishes to for stenographic reasons, can send dummy
data by itself. Also it will handle all necessary parsing and encoding
for whichever protocol it is trying to mimic. It creates its own
threads to copy data from freenet and send it.
For a stream-based plugin, it runs its own loop of getting/writing
data from/to StreamConnectionHandler(node-side), encoding/decoding,
and sending/receiving to its socket. It must have its own flow control
mechanism.
To be discussed-
One major point I haven't discussed is how multiple transports work.
For e.g. a node might be able to communicate using several transports,
while its peer might not be able to do so. It had been the case that
for datagrams one needed a single port that sent messages to all its
clients. But with multiple transports, the node needs a record of
active transports for each peer. Apart from this we need to discuss
how the messages are distributed between transports. This also means
some transports might be slow, or the user prefers one of the
transports. Although I wish to address it within the timeline of GSoC,
I am not sure how it will turn out. But for the moment my idea is to
be able to choose the needed transport, although the framework to use
them simultaneously or switch from one to another will be supported.
If I find that the logic to implement selection of transports is easy,
then I can implement it. But I assume that needs some usage analysis
as well. Based on that we can comment.
My objective is to atleast get a functional freenet that will atleast
figure one way to connect to a peer, and this can be different for
different peers.
I need your opinions on this. But atleast some modification of
PeerNode to keep track of which transport works is a must.
This is a haphazard report, that has questions, issues as well as
conclusions. Based on feedback and suggestions, for the next two weeks
I wish to work on refactoring changes to support the above. Also
thanks to toad for making it very simple for me, and the fact that
over a month, all his design implementation ideas mentioned on email
and irc are consistent and complementary to each other.
I will work on a separate branch and I am not sure how it follows from
now on, as I submit code. The present idea is to create a branch for
every task, to be completed every week. Comments and suggestions are a
must!
Regards,
Chetan Hosmani
RSS Feed