[statechart] [spirit] OSI delimited message framing
Hello, I need to design a delimited message framing for TCP/IP sockets, probably using Asio to facilitate transport. Then build an OSI layer 7 application vernier to handle framing. I've dabbled a bit with Spirit and cooked up a couple of "simple" micro grammars throughout the code base. In a sense, I could see a delimited message framing strategy being a grammar of sorts, but this seems somewhat complex for a grammar to handle when you consider the overhead of verifying pass/fail (message or not a message), plus extracting the message within the delimiters. Or, more likely approach the problem with statechart in mind. I've dealt with framed messages in the past more like this. One state is the escape byte, which can be followed by SOM, EOM, or another true-escape-byte (intended message byte). Plus other fields like message type, sub-type, how to handle payload, arguments, and so forth. Anyone else dealt with things like this before? With Spirit? With Statechart? Regards, Michael Powell
On 06/19/13 06:55, Michael Powell wrote:
Hello,
I need to design a delimited message framing for TCP/IP sockets, probably using Asio to facilitate transport. Then build an OSI layer 7 application vernier to handle framing.
I've dabbled a bit with Spirit and cooked up a couple of "simple" micro grammars throughout the code base. In a sense, I could see a delimited message framing strategy being a grammar of sorts, but this seems somewhat complex for a grammar to handle when you consider the overhead of verifying pass/fail (message or not a message), plus extracting the message within the delimiters.
Or, more likely approach the problem with statechart in mind. I've dealt with framed messages in the past more like this. One state is the escape byte, which can be followed by SOM, EOM, or another true-escape-byte (intended message byte). Plus other fields like message type, sub-type, how to handle payload, arguments, and so forth.
Anyone else dealt with things like this before? With Spirit? With Statechart?
Or with Lex? I would think using just the lexer would work, since, IIRC, lexers are a type of FSM, and I that's what StateChart is also. OTOH, since you have to send and receive over a network. Wouldn't karma be a candidate for composing the message just as Spirit would be a candidate for reading the message?
Regards,
Michael Powell
On 06/19/2013 01:55 PM, Michael Powell wrote:
I need to design a delimited message framing for TCP/IP sockets, probably using Asio to facilitate transport. Then build an OSI layer 7 application vernier to handle framing.
I have a messaging middleware prototype that is build on top of Asio at: http://axiomq.sourceforge.net/ There is no documentation yet, as I have not settled on the design yet. The prototype has a basic_message_socket class with two functions async_read_message() and async_send_message() that handles the exchange of messages. A concrete implementation of basic_message_socket is needed to support a specific protocol. I have a very basic and feature-weak implementation of ZTMP/2.0 (the ZeroMQ protocol.) It currently uses a home-brewn state machine to parse the protocol. I have not yet examined if this could be replaced by Spirit, StateChart, or MSM. A connection can be established with async_connect() and async_accept(). The latter is in the basic_message_acceptor class. These functions only report back to the application once the two sides have exchanged protocol headers and determined if they are compatible.
On 06/19/2013 04:55 AM, Michael Powell wrote:
Hello,
I need to design a delimited message framing for TCP/IP sockets, probably using Asio to facilitate transport. Then build an OSI layer 7 application vernier to handle framing.
I've dabbled a bit with Spirit and cooked up a couple of "simple" micro grammars throughout the code base. In a sense, I could see a delimited message framing strategy being a grammar of sorts, but this seems somewhat complex for a grammar to handle when you consider the overhead of verifying pass/fail (message or not a message), plus extracting the message within the delimiters.
Or, more likely approach the problem with statechart in mind. I've dealt with framed messages in the past more like this. One state is the escape byte, which can be followed by SOM, EOM, or another true-escape-byte (intended message byte). Plus other fields like message type, sub-type, how to handle payload, arguments, and so forth.
Anyone else dealt with things like this before? With Spirit? With Statechart?
Hi Michael - We write a lot of communication layers for both internal and client use. One approach is to use use Spirit Qi/Karma to act as a translator. It takes data objects in and produces byte streams (Karma) or takes a byte stream in and produces objects (Qi). We use Asio for the communication layer and Spirit for the protocol layer. This can all be nicely wrapped up in a generic objects. Spirit is stack based and so interrupting mid stream and saving state when partial messages are received can be tricky. The *answer* is to use a coroutine (Boost.Context). We have also used a statemachine approach but prefer to use MSM for protocol machines. It is much faster. Just to add a bit more to the discussion: message = omit[ *( !som >> byte_ ) ] // sync stream to start of msg >> som // start of msg rule >> *( byte_ - eom ) // put your rule here >> eom // read end of msg ; The version above could synthesize a std::vector<unit8_t> for example. You could have it parse the various message types or some generic structure (16-bit command followed by N-byte payload). I hope this gives you some thoughts. michael -- Michael Caisse ciere consulting ciere.com
Thank y'all for the feedback. Will probably go with some sort of context for a byte-wise unwrap operation (decouples from whatever read source), and spits out a smart pointer next message when it can be unwrapped. I gather here, bind to whatever stateful context handler needs to be used for the next character. Simple enough. For the wrapping side of things, I'd like to go with karma if I can. The wrapping sequence is one layer in the OSI layer 7 protocol. Loosely coupled from whatever message payloading. SOM and EOM sequences are straightforward enough. However, I am unclear how to conditionally generate the next byte in the sequence. In other words, would be either the next byte as-is, or an escape delimited byte. Loosely: typedef std::vector<uint8_t> buffer_type; byte next_ = some_value; karma::rule<buffer_type::iterator> dle_ %= next_ | ( byte_(m_dle) << next_ ); However, how to tell karma what the condition is? On Wed, Jun 19, 2013 at 12:27 PM, Michael Caisse <mcaisse-lists@ciere.com> wrote:
On 06/19/2013 04:55 AM, Michael Powell wrote:
Hello,
I need to design a delimited message framing for TCP/IP sockets, probably using Asio to facilitate transport. Then build an OSI layer 7 application vernier to handle framing.
I've dabbled a bit with Spirit and cooked up a couple of "simple" micro grammars throughout the code base. In a sense, I could see a delimited message framing strategy being a grammar of sorts, but this seems somewhat complex for a grammar to handle when you consider the overhead of verifying pass/fail (message or not a message), plus extracting the message within the delimiters.
Or, more likely approach the problem with statechart in mind. I've dealt with framed messages in the past more like this. One state is the escape byte, which can be followed by SOM, EOM, or another true-escape-byte (intended message byte). Plus other fields like message type, sub-type, how to handle payload, arguments, and so forth.
Anyone else dealt with things like this before? With Spirit? With Statechart?
Hi Michael -
We write a lot of communication layers for both internal and client use.
One approach is to use use Spirit Qi/Karma to act as a translator. It takes data objects in and produces byte streams (Karma) or takes a byte stream in and produces objects (Qi). We use Asio for the communication layer and Spirit for the protocol layer. This can all be nicely wrapped up in a generic objects.
Spirit is stack based and so interrupting mid stream and saving state when partial messages are received can be tricky. The *answer* is to use a coroutine (Boost.Context).
We have also used a statemachine approach but prefer to use MSM for protocol machines. It is much faster.
Just to add a bit more to the discussion:
message = omit[ *( !som >> byte_ ) ] // sync stream to start of msg >> som // start of msg rule >> *( byte_ - eom ) // put your rule here >> eom // read end of msg ;
The version above could synthesize a std::vector<unit8_t> for example. You could have it parse the various message types or some generic structure (16-bit command followed by N-byte payload).
I hope this gives you some thoughts. michael
-- Michael Caisse ciere consulting ciere.com _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
Depending on your framework - if you use a std::stream from which you get your sequence of bytes which will be parsed in order to extract a message the simple example from boost.coroutine docu could help. It demonstrates how a execution is suspended if the std::stream (stream_buf) can not provide enough bytes.
participants (1)
-
Bjorn Reese -
Larry Evans -
Michael Caisse -
Michael Powell -
Oliver Kowalke