Tools.h++ Manual

104011 Tandem Computers Incorporated 21-221
21
RWWTokenizer
Synopsis #include <rw/wtoken.h>
RWWString str("a string of tokens", RWWString::ascii);
RWWTokenizer(str); // Lex the above string
Description Class RWWTokenizer is designed to break a string up into separate tokens,
delimited by an arbitrary “white space”. It can be thought of as an iterator for
strings and as an alternative to the C library function wstok () which has the
unfortunate side effect of changing the string being tokenized.
Example
Program output:
Something
is
rotten
in
the
state
of
Denmark
Public constructor RWWTokenizer(const RWWString& s);
Construct a tokenizer to lex the string s .
#include <rw/wtoken.h>
#include <rw/rstream.h>
main()
{
RWWString a(L"Something is rotten in the state of Denmark");
RWWTokenizer next(a); // Tokenize the string a
RWWString token; // Will receive each token
// Advance until the null string is returned:
while (!(token=next()).isNull())
cout << token << "\n";
}