Module scan

A Lexical Scanner.

Lexical scanners are a smarter and cleaner alternative to the primitive strtok function. Each time you call scan_next, the scanner finds the next token,

ScanState *ts = scan_new_from_string("hello = (10,20,30)"));
scan_next(ts);
char *name = scan_get_str(ts);  // will be "hello"
char ch = scan_next(ts);  // will be '='
scan_next(ts);  // skip '('
scan_next(ts);
double val1 = scan_get_number(ts);  // 10
scan_next(ts); // skip ','
double val2 = scan_get_number(ts); // 20

At any point, ts->type tells you the next available token. Note that by default this scanner ignores space.

A convenient higher-level function is scan_scanf; the equivalent of above code is simply:

scan_scanf(ts,"%s %c (%f,%f",&name,&ch,&val1,&val2).

See test-scan.c for examples of various uses.

Tables

ScanState Scanner type.

Configuration

scan_set_flags (ts, flags) set flags.
scan_set_line_comment (ts, cc) line comment (either one or two characters).
scan_force_line_mode (ts) tell the scanner not to grab the next line automatically.
scan_push_back (ts) tell the scanner not to advance on following scan_next.

Constructing

scan_new_from_string (str) scanner from a string.
scan_new_from_file (fname) scanner from a file.
scan_new_from_stream (stream) scanner from an existing file stream.

Grabbing

scan_fetch_line (ts, skipws) fetch a new line from the stream, if defined.
scan_getch (ts) get the next character.
scan_advance (ts, offs) Move the scan reader position directly with an offset.
scan_peek (ts, offs) look at character ahead
scan_get_upto (ts, target, buff, bufsz) grab a string upto (but not including) a final target string.
scan_numbers (ts, values, sz) grab up to sz numbers from the stream.

Skipping

scan_skip_whitespace (ts) skip white space, reading new lines if necessary.
scan_skip_space (ts) skip white space and single-line comments.
scan_skip_digits (ts) skip digits.
scan_skip_until (ts, type) skip until a token is found with type.
scan_next_number (ts, val) fetch the next number, skipping any other tokens.
scan_next_iden (ts, buff, len) fetch the next word, skipping other tokens.
scan_next_item (ts, type, buff, sz) fetch the next item, skipping other tokens.

Scanning

scan_next (ts) advance to the next token.

Getting

scan_get_tok (ts, tok, len) copy the current token to a buff.
scan_get_str (ts) get current token as string.
scan_scanf (ts, fmt, ...) Formatted reading from the scanner, like scanf.
scan_get_line (ts, buff, len) get the rest of the current line.
scan_next_line (ts) fetch the next line and force line mode.
scan_get_number (ts) get the current token as a number.


Tables

ScanState
Scanner type.

Fields:

  • line int current line in file, if not just parsing a string.
  • type int One of the following:

    T_END, T_EOF=0, T_TOKEN, T_IDEN=1, T_NUMBER, T_STRING, T_CHAR, T_NADA

  • int_type int One of

    T_DOUBLE, T_INT, T_HEX, T_OCT,

Configuration

scan_set_flags (ts, flags)

set flags.

  • C_IDEN words may contain underscores
  • C_NUMBER instead of T_NUMBER, return T_INT,T_HEX and T_DOUBLE
  • C_STRING parse C string escapes
  • C_WSPACE don’t skip whitespace

Parameters:

scan_set_line_comment (ts, cc)
line comment (either one or two characters).

Parameters:

scan_force_line_mode (ts)
tell the scanner not to grab the next line automatically.

Parameters:

scan_push_back (ts)
tell the scanner not to advance on following scan_next.

Parameters:

Constructing

scan_new_from_string (str)
scanner from a string.

Parameters:

  • str const char *

Returns:

    ScanState *
scan_new_from_file (fname)
scanner from a file.

Parameters:

  • fname const char *

Returns:

    ScanState *
scan_new_from_stream (stream)
scanner from an existing file stream.

Parameters:

  • stream FILE *

Returns:

    ScanState *

Grabbing

scan_fetch_line (ts, skipws)
fetch a new line from the stream, if defined. Advances the line count – not used if the scanner has been given a string directly.

Parameters:

Returns:

    bool
scan_getch (ts)
get the next character.

Parameters:

Returns:

    char
scan_advance (ts, offs)
Move the scan reader position directly with an offset.

Parameters:

scan_peek (ts, offs)
look at character ahead

Parameters:

Returns:

    char
scan_get_upto (ts, target, buff, bufsz)
grab a string upto (but not including) a final target string. Advances the scanner (use scan_advance with negative offset to back off)

Parameters:

  • ts ScanState *
  • target const char *
  • buff char *
  • bufsz int

Returns:

    int
scan_numbers (ts, values, sz)
grab up to sz numbers from the stream. scan_next_line can be used to limit this to the current line only.

Parameters:

Returns:

    int

Skipping

scan_skip_whitespace (ts)
skip white space, reading new lines if necessary.

Parameters:

Returns:

    bool
scan_skip_space (ts)
skip white space and single-line comments.

Parameters:

scan_skip_digits (ts)
skip digits.

Parameters:

scan_skip_until (ts, type)
skip until a token is found with type. May return false if the scanner ran out.

Parameters:

Returns:

    bool
scan_next_number (ts, val)
fetch the next number, skipping any other tokens.

Parameters:

Returns:

    bool
scan_next_iden (ts, buff, len)
fetch the next word, skipping other tokens.

Parameters:

Returns:

    char *
scan_next_item (ts, type, buff, sz)
fetch the next item, skipping other tokens.

Parameters:

Returns:

    bool

Scanning

scan_next (ts)
advance to the next token. Usually this skips whitespace, and single-line comments if defined.

Parameters:

Returns:

    ScanTokenType

Getting

scan_get_tok (ts, tok, len)
copy the current token to a buff.

Parameters:

Returns:

    char *
scan_get_str (ts)
get current token as string.

Parameters:

Returns:

    char *
scan_scanf (ts, fmt, ...)

Formatted reading from the scanner, like scanf. Flags start with ‘%’, and ‘%%’ encodes a literal ‘%’.

  • v value
  • s identifier
  • l rest of line
  • q quoted string
  • i int
  • f double
  • c char
  • . don’t care!

Parameters:

Returns:

    bool
scan_get_line (ts, buff, len)
get the rest of the current line. This trims any leading whitespace.

Parameters:

Returns:

    char *
scan_next_line (ts)
fetch the next line and force line mode. After this, the scanner will regard end-of-line as end of input.

Parameters:

Returns:

    const char *
scan_get_number (ts)
get the current token as a number.

Parameters:

Returns:

    double
generated by LDoc 1.4.3 Last updated 2015-04-27 12:35:55