Here you are: typedef std::basic_string<TCHAR> tstring_t; typedef boost::char_separator<TCHAR> Delimiter_t; typedef boost::tokenizer< Delimiter_t, tstring_t::const_iterator, tstring_t

TabTokenizer_t;

Delimiter_t csDelimiter(#my_delimeter_tchar#); TabTokenizer_t tTokens(#my_input_tchar_string#, csDelimiter); TabTokenizer_t::iterator itTok(tTokens.begin()); TabTokenizer_t::iterator itTokEnd(tTokens.end()); while (itTok != itTokEnd) { //do stuff } This will tokenize an input tstring_t. I used TCHAR because they are translated as char or wchar_t depending if Unicode is turned on or not. If you are only interested in wide char strings then used wchar_t directly. -delfin

-----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users- bounces@lists.boost.org] On Behalf Of Rozental, Gennadiy Sent: Monday, September 20, 2004 8:59 AM To: 'boost-users@lists.boost.org' Subject: RE: [Boost-users] Can boost::tokenizer tokenize 2byte character string?

If it 2byte why are you using boost::char_separator<char>?

Shouldn't you be using boost::char_separator<wchar_t>?

Gennadiy.

...

-----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Lee, Joo-Young Sent: Saturday, September 18, 2004 12:39 AM To: boost-users@lists.boost.org Subject: [Boost-users] Can boost::tokenizer tokenize 2byte character string?

Hi.

I try to use 'boost::tokenizer<boost::char_separator<char> >' to separate 2byte character string like Korean, Japanese, or Chinese.

But, I found that it does not works correctly.

Is there a solution?

Thanks for the help,

Lee Joo-Young

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users