Penlight

• Index

Contents

• Functions

Libraries

• pl
• pl.Set
• pl.app
• pl.array2d
• pl.class
• pl.compat
• pl.comprehension
• pl.config
• pl.data
• pl.dir
• pl.file
• pl.func
• pl.import_into
• pl.input
• pl.lapp
• pl.lexer
• pl.luabalanced
• pl.operator
• pl.path
• pl.permute
• pl.pretty
• pl.seq
• pl.sip
• pl.strict
• pl.stringio
• pl.stringx
• pl.tablex
• pl.template
• pl.test
• pl.text
• pl.types
• pl.url
• pl.utils
• pl.xml

Classes

• pl.Date
• pl.List
• pl.Map
• pl.MultiMap
• pl.OrderedMap

Manual

• Introduction
• Tables and Arrays
• Strings. Higher-level operations on strings.
• Paths and Directories
• Date and Time
• Data
• Functional Programming
• Additional Libraries
• Technical Choices

Examples

• seesubst.lua
• sipscan.lua
• symbols.lua
• test-cmp.lua
• test-data.lua
• test-listcallbacks.lua
• test-pretty.lua
• test-symbols.lua
• testapp.lua
• testclone.lua
• testconfig.lua
• testglobal.lua
• testinputfields.lua
• testinputfields2.lua
• testxml.lua
• which.lua

Module pl.lexer

Lexical scanner for creating a sequence of tokens from text.

lexer.scan(s) returns an iterator over all tokens found in the string s. This iterator returns two values, a token type string (such as ‘string’ for quoted string, ‘iden’ for identifier) and the value of the token.

Versions specialized for Lua and C are available; these also handle block comments and classify keywords as ‘keyword’ tokens. For example:

> s = 'for i=1,n do'
> for t,v in lexer.lua(s)  do print(t,v) end
keyword for
iden    i
=       =
number  1
,       ,
iden    n
keyword do

See the Guide for further discussion

Functions

scan (s, matches[, filter[, options]])	create a plain token iterator from a string or file-like object.
insert (tok, a1, a2)	insert tokens into a stream.
getline (tok)	get everything in a stream upto a newline.
lineno (tok)	get current line number.
getrest (tok)	get the rest of the stream.
get_keywords ()	get the Lua keywords as a set-like table.
lua (s[, filter[, options]])	create a Lua token iterator from a string or file-like object.
cpp (s[, filter[, options]])	create a C/C++ token iterator from a string or file-like object.
get_separated_list (tok[, endtoken=')'[, delim=']])	get a list of parameters separated by a delimiter from a stream.
skipws (tok)	get the next non-space token from the stream.
expecting (tok, expected_type, no_skip_ws)	get the next token, which must be of the expected type.

Functions

scan (s, matches[, filter[, options]])

create a plain token iterator from a string or file-like object.

Parameters:

• s string or file a string or a file-like object with :read() method returning lines.
• matches tab an optional match table - array of token descriptions. A token is described by a {pattern, action} pair, where pattern should match token body and action is a function called when a token of described type is found.
• filter tab a table of token types to exclude, by default {space=true} (optional)
• options tab a table of options; by default, {number=true,string=true}, which means convert numbers and strip string quotes. (optional)

insert (tok, a1, a2)

insert tokens into a stream.

Parameters:

• tok a token stream
• a1 a string is the type, a table is a token list and a function is assumed to be a token-like iterator (returns type & value)
• a2 string a string is the value

getline (tok)

get everything in a stream upto a newline.

Parameters:

• tok a token stream

Returns:

a string

lineno (tok)

get current line number.

Parameters:

• tok a token stream

Returns:

the line number. if the input source is a file-like object, also return the column.

getrest (tok)

get the rest of the stream.

Parameters:

• tok a token stream

Returns:

a string

get_keywords ()

get the Lua keywords as a set-like table. So res["and"] etc would be true.

Returns:

a table

lua (s[, filter[, options]])

create a Lua token iterator from a string or file-like object. Will return the token type and value.

Parameters:

• s string the string
• filter tab a table of token types to exclude, by default {space=true,comments=true} (optional)
• options tab a table of options; by default, {number=true,string=true}, which means convert numbers and strip string quotes. (optional)

cpp (s[, filter[, options]])

create a C/C++ token iterator from a string or file-like object. Will return the token type type and value.

Parameters:

• s string the string
• filter tab a table of token types to exclude, by default {space=true,comments=true} (optional)
• options tab a table of options; by default, {number=true,string=true}, which means convert numbers and strip string quotes. (optional)

get_separated_list (tok[, endtoken=')'[, delim=']])

get a list of parameters separated by a delimiter from a stream.

Parameters:

• tok the token stream
• endtoken string end of list. Can be ‘\n’ (default ')')
• delim string separator (default ')

Returns:

a list of token lists.

skipws (tok)

get the next non-space token from the stream.

Parameters:

• tok the token stream.

expecting (tok, expected_type, no_skip_ws)

get the next token, which must be of the expected type. Throws an error if this type does not match!

Parameters:

• tok the token stream
• expected_type string the token type
• no_skip_ws bool whether we should skip whitespace