# KEHOME/doc/KRtoken.txt
# Dec/22/2002

#========================#
# MKR tokens             #
# version 4.5            #
#                        #
# Richard H. McCullough  #
#========================#

# The MKR parser has three phases
#
# 1. token.icn: get_word(): identify low-level word classes, e.g.
#	b	white space
#	s	separator
#	w	word
# 2. token.icn: get_token(): identify special words and operators, e.g.
#	A	at
#	H	has isin
#	D	do
#	o	od
#	d	done
#	R	relation: isa isa* isa**n ...
#	P	preposition: of with from to out
#	=	assignment operator: =,+=,-=,*=
# 3. symbol.icn: get_symbol(): identify phrases, sentences, ..., e.g.
#	www	word phrase
#	wHw=w	subject has attribute=value
#	wRw	subject rel object
#	wiwPw=w	subject is  object with attribute=value


#------------------------------------#
# Here's the complete list of tokens #
#------------------------------------#

B	white space: blank tab return newline newpage
N	non separator: ~ / \ ? . $  (included in w)
W	weak separator: + - * :     (included in x)
S	strong separator: , ; = ! & | [ ] { } ( ) < > " ' #
w	word: consecutive bytes not B nor S nor W
x	extended word: consecutive w or W

"	quote mark: " or '
q	quote
#	comment mark: #
c	comment

# NOTE:
# For high level parsing (phase 3 in symbol.icn),
# the types above are remapped to
#	B	=> deleted
#	W	=> individual characters
#	S	=> individual characters
#	w	=> w
#	x	=> w
#	"	=> not used
#	q	=> w
#	#	=> not used
#	c	=> deleted


,	list separator: phrase
;	list separator: sentence, proposition

~	OK in word: not
:	OK in word: view-concept, class-number, unit
+	OK in word
-	OK in word
*	OK in word: wildcard

A	at: context -- space,time,view
C	changes: event-eobject
D	do do* vdo
d	done
E	else
F	fi
G	group quantifier: all any either
H	has isin
h	HTML command
I	if
i	is isa iss isu isd isi
J	conjunction: ";" iff implies causes means
j	conjunction: and or
N	complement: not non
P	preposition: out of with from to od at
p	haspart, isapart
Q	quantifier: a some the no
R	relation: isa* isa** ... isc isc* isc** ... in ...
S	assignment: set vset unset
T	then
U	unknown
W	generator: every when while
Y	group definition: begin
Z	group definition: end
z	exit

()	grouping: not currently used
[]	begin/end of comma list: phrase
{}	begin/end of semicolon list: proposition
<>	embedded HTML

/	down one level in hierarchy
\	up one level in hierarchy
?	question
!	sh command
$	value of variable
$()	product (a la KornShell) -- not currently used
..	dot variable: current genus
.	dot variable: current concept
...	dot variable: current species

=	assignment operator =
=	Set add operator +=
=	Set delete operator -=
=	Set intersection operator *=
=	production operator :=
=	Backus-Naur Form grammar operator ::=


&	and, asynchronous execution (not implemented)
|	or,  pipe (not implemented)
~	not, login directory


##### possible additions #####
g	group: concept directory relation Set list sequence hierarchy lattice
h	hierarchy: the word
m	method
o	od: direct object
r	relation: the word
s	space
t	time
u	unit
v	view
