Re: Network library: What we have until now
Hi Boris,
I updated
http://www.crystalclearsoftware.com/cgi-bin/boost_wiki/wiki.pl?BoostNet
to document why what decisions were made. We have until now:
* The network library should support four I/O models which are known to many programmers as blocking, non-blocking, multiplexing and asynchronous.
Personally, I don't think the multiplexing interface is appropriate or necessary. Essentially the same style can be achieved with multi-threaded async and a boos::function queueing solution. In other words, it is not a group of network objects that is being manipulated. It is a portable dispatcher that gets boost::function<>'s queued to it. It therefore suffers none of the portability issues that a direct wrapper to select would have. I also believe that non-blocking is an I/O only thing; it is not needed for connect or accept.
* It should be possible to switch between I/O models at runtime (which means you have one socket class supporting all I/O models instead of different socket classes each of them supporting only one I/O model).
Perhaps "switch" isn't the best term. On a per-operation basis, the user should chose between styles.
* There should be an asynchronous I/O library as other libraries might want to do asynchronous I/O, too.
I don't see this as practical. There is no such thing as async I/O that can be portably described. This means that each kind of async object has to provide the right implementation. Perhaps the scope of portability needs to be stated and agreed upon. When I implemented async pipe I/O on Windows NT/9x, I could not use the same code for I/O as I did with sockets or as I would for files (subtle differences). I think a common pattern for async behavior should be agreed upon and if some implementation details turn out to be reusable, that's a good happenstance. My prediction is that on Unix platforms, there will be more to reuse between sockets, pipes, files, etc.. On Windows, there will be little.
* There should be an asynchronous design pattern which should be used by all Boost libraries which support asynchronous operations.
I agree completely on this one :)
* On a low level the network library should be close to what is known as Berkeley sockets to many programmers.
I think sockets should be hidden. They are not universally used, for example Mac OS9 doesn't and I just assume others exist. They don't have any magical powers; they are just a C abstraction. A C++ abstraction should replace them.
* On a high level there should be I/O streams support.
I agree again ;)
As far as I can see from all the discussions that's what we agree on. There have been some ideas like event handlers and dispatchers but I am not sure where to put them - have a look at http://www.highscore.de/boost/net/packages.png please (which is also in the Wiki page). This is the overall architecture of the network library. If you want me to change or add something please tell me.
We have a couple straw men started (Michel's and mine). Perhaps it is just as good to start just throwing darts at them? I could probably have a full implementation of the interfaces I propose in a small number of weeks (2-3). Having done it once before really helps<g>, but I wanted to get to some common ground on the philosophy before going to that step. The philosophy I have on this may be out of sync with some (pun intended), but IMHO, we need to hash over the philosophy as much as anything. We have a lot of folks that are very comfortable with the sockets interface, but I don't believe that exposing its idiosyncrasies is the right approach. Everyone that writes sockets code runs into the same problems, and solves them in varying degrees of sophistication and completeness. A robust solution is not as difficult as some might think. It's certainly easier than getting agreement on philosophical matters. ;) Best regards, Don __________________________________ Do you Yahoo!? Yahoo! Small Business - Try our new resources site! http://smallbusiness.yahoo.com/resources/
Don G wrote:
Hi Boris,
I updated
http://www.crystalclearsoftware.com/cgi-bin/boost_wiki/wiki.pl?BoostNet
to document why what decisions were made. We have until now:
* The network library should support four I/O models which are known to many programmers as blocking, non-blocking, multiplexing and asynchronous.
Personally, I don't think the multiplexing interface is appropriate or necessary. Essentially the same style can be achieved with multi-threaded async and a boos::function queueing solution. In other words, it is not a group of network objects that is being manipulated. It is a portable dispatcher that gets boost::function<>'s queued to it. It therefore suffers none of the portability issues that a direct wrapper to select would have.
The multiplexing interface can be used in a single-threaded application while the async stuff will be multi-threaded. There might be network developers (like me for example :-) who don't want to synchronize callbacks but just want to build the whole application around a blocking call to select (or any other multiplexing operation).
[...]
* There should be an asynchronous I/O library as other libraries might want to do asynchronous I/O, too.
I don't see this as practical. There is no such thing as async I/O that can be portably described. This means that each kind of async object has to provide the right implementation. Perhaps the scope of portability needs to be stated and agreed upon. When I implemented async pipe I/O on Windows NT/9x, I could not use the same code for I/O as I did with sockets or as I would for files (subtle differences).
Basically I like the idea to have an async I/O library just like std::iostreams which supports synchronous I/O. However I don't know how difficult it is to implement such a library but isn't POSIX aio exactly about this?
[...]
* On a low level the network library should be close to what is known as Berkeley sockets to many programmers.
I think sockets should be hidden. They are not universally used, for example Mac OS9 doesn't and I just assume others exist. They don't have any magical powers; they are just a C abstraction. A C++ abstraction should replace them.
Opinions seem to vary on this. When I once proposed to build a C++ network abstraction with I/O streams I got complaints that it should be possible to use sockets directly in a lower level. Whatever you write the other group will join the thread and complain. :)
[...] We have a couple straw men started (Michel's and mine). Perhaps it is just as good to start just throwing darts at them? I could probably have a full implementation of the interfaces I propose in a small number of weeks (2-3). Having done it once before really helps<g>, but I wanted to get to some common ground on the philosophy before going to that step.
Yes, that's why I am trying to find a design the majority agrees with. If we end up with several network libraries implemented by different people noone will do the effort to understand and compare design decisions.
[...] of sophistication and completeness. A robust solution is not as difficult as some might think. It's certainly easier than getting agreement on philosophical matters. ;)
Absolutely, as we can see. :) Boris
Boris wrote:
I think sockets should be hidden. They are not universally used, for example Mac OS9 doesn't and I just assume others exist. They don't have any magical powers; they are just a C abstraction. A C++ abstraction should replace them.
Opinions seem to vary on this. When I once proposed to build a C++ network abstraction with I/O streams I got complaints that it should be possible to use sockets directly in a lower level. Whatever you write the other group will join the thread and complain. :)
As I recollect the discussions and the general opinions there has been some agreement that at level 1 there should be a thin wrapper on top of the socket api since a lot of developers knows these apis and it would be beneficial to have them in a C++ wrapper (with exceptions, RAII and a more modern interface). /Michel
On 4/13/05, Don G <dongryphon@yahoo.com> wrote:
I also believe that non-blocking is an I/O only thing; it is not needed for connect or accept.
Not sure about the need for a non-blocking accept, but a non-blocking connect is a must-have. -- Caleb Epstein caleb dot epstein at gmail dot com
Caleb Epstein wrote:
On 4/13/05, Don G <dongryphon@yahoo.com> wrote:
I also believe that non-blocking is an I/O only thing; it is not needed for connect or accept.
Not sure about the need for a non-blocking accept, but a non-blocking connect is a must-have.
I second that. :-)
Caleb Epstein wrote:
On 4/13/05, Don G <dongryphon@yahoo.com> wrote:
I also believe that non-blocking is an I/O only thing; it is not needed for connect or accept.
Not sure about the need for a non-blocking accept, but a non-blocking connect is a must-have.
While a non-blocking connect is a must-have I think a non-blocking accept is essential also. A server needs to be just as responsive to other things happening as a client. One might need to pause or stop the server completely, depending on the meaning given to each, and a server might be accepting connections on more than one endpoint. Without a non-blocking accept, both these general possibilities become impossible and one ends up with a single end-point server which must be killed to end the process, which I do not think scales well in today's world of network programming.
----- Original Message ----- From: "Edward Diener" <eddielee@tropicsoft.com> To: <boost@lists.boost.org> Sent: Thursday, April 14, 2005 4:57 AM Subject: [boost] Re: Re: Network library: What we have until now
Caleb Epstein wrote:
On 4/13/05, Don G <dongryphon@yahoo.com> wrote:
I also believe that non-blocking is an I/O only thing; it is not needed for connect or accept.
Not sure about the need for a non-blocking accept, but a non-blocking connect is a must-have.
While a non-blocking connect is a must-have I think a non-blocking accept is essential also. A server needs to be just as responsive to other things happening as a client. One might need to pause or stop the server completely, depending on the meaning given to each, and a server might be accepting connections on more than one endpoint. Without a non-blocking accept, both these general possibilities become impossible and one ends up with a single end-point server which must be killed to end the process, which I do not think scales well in today's world of network programming.
Asynchronous listen, accept and connect please. Network activity is innately asynchronous and a design for *the* network library that veers away from that is precluding what is more natural. A hybrid of async (e.g. accept) and sync (e.g. connect) also results in a more complex set of classes+templates. Especially when compared with a design where the lowest (i.e. base) class related to clients and servers is one and the same. Both ends of a connection-oriented service issue network requests (connect or listen) and subsequently wait for establishment. This commonality of operation can be exploited nicely. For those still wanting synchronous behaviour (there are perfectly good reasons I'm sure ;-) this can still be achieved over the top of async. The same cannot be said for the reverse. These points are relevant to I/O of course.
On Tue, 12 Apr 2005 21:39:24 -0700 (PDT), Don G <dongryphon@yahoo.com> wrote:
* The network library should support four I/O models which are known to many programmers as blocking, non-blocking, multiplexing and asynchronous.
Personally, I don't think the multiplexing interface is appropriate or necessary.
Rather it's vital.
Essentially the same style can be achieved with multi-threaded async and a boos::function queueing solution. In other words, it is not a group of network objects that is being manipulated. It is a portable dispatcher that gets boost::function<>'s queued to it. It therefore suffers none of the portability issues that a direct wrapper to select would have.
I also believe that non-blocking is an I/O only thing; it is not needed for connect or accept.
None of the calls should block. Otherwise you won't be able to build single threaded applications with select/poll/epoll-based loops. []
* There should be an asynchronous design pattern which should be used by all Boost libraries which support asynchronous operations.
I agree completely on this one :)
I hope this won't happen. There are several I/O patterns in use out there.
* On a low level the network library should be close to what is known as Berkeley sockets to many programmers.
I think sockets should be hidden. They are not universally used, for example Mac OS9 doesn't and I just assume others exist. They don't have any magical powers; they are just a C abstraction.
Was not Mac OS Classic been buried long ago? IMO, sockets are abstractions out of the scope of any language. Their interfaces can be implemented using any general purpose programming language, but still you'll have to call an underlying OS C API. That's a fact of life and I don't think this is bad - C is the most portable programming language ever.
A C++ abstraction should replace them.
With this kind of attitude you might end up rewritting the whole world you've been living in <g>
* On a high level there should be I/O streams support.
I agree again ;)
What kind of streams? If those are std::iostream's, than that is a bad idea. Binary read/write like interface will suffice and can be trivially wrapped in std::streambuf -> std::iostream. []
The philosophy I have on this may be out of sync with some (pun intended), but IMHO, we need to hash over the philosophy as much as anything. We have a lot of folks that are very comfortable with the sockets interface, but I don't believe that exposing its idiosyncrasies is the right approach. Everyone that writes sockets code runs into the same problems, and solves them in varying degrees of sophistication and completeness. A robust solution is not as difficult as some might think. It's certainly easier than getting agreement on philosophical matters. ;)
Here is my two philosophy cents. You propose another set of concepts over socket concepts just to send/receive a bunch of mere bytes. Geez, another glue layer over sockets that adds nothing but syntax sugar. Aren't we already tired of software bloated from glue layers over glue layers over...? You don't just send bytes, rather you execute protocols. Sockets have the right implementation level and complexity to build *efficient* protocols upon. Protocol is the only glue layer you really always need, the layer that binds your application logic with network transport layer (sockets) providing the right C++ or whatever interface. -- Maxim Yegorushkin
participants (1)
-
Boris -
Caleb Epstein -
Don G -
Edward Diener -
Maxim Yegorushkin -
Michel André -
Peter Dimov -
Scott Woods