5 Commits

Author	SHA1	Message	Date
Chris Lattner	5a7971e0c3	This change refactors some of the low-level lexer interfaces a bit. ... Token now has a class of kinds for "literals", which include numeric constants, strings, etc. These tokens can optionally have a pointer to the start of the token in the lexer buffer. This makes it faster to get spelling and do other gymnastics, because we don't have to go through source locations. This change is performance neutral, but will make other changes more feasible down the road. llvm-svn: 63028	2009-01-26 19:29:26 +00:00
Chris Lattner	08354fef13	add a simplified lexer ctor that sets up the lexer to raw-lex an ... entire file. llvm-svn: 62414	2009-01-17 07:35:14 +00:00
Chris Lattner	d32480d3db	this massive patch introduces a simple new abstraction: it makes ... "FileID" a concept that is now enforced by the compiler's type checker instead of yet-another-random-unsigned floating around. This is an important distinction from the "FileID" currently tracked by SourceLocation. *That* FileID may refer to the start of a file or to a chunk within it. The new FileID *only* refers to the file (and its #include stack and eventually #line data), it cannot refer to a chunk. FileID is a completely opaque datatype to all clients, only SourceManager is allowed to poke and prod it. llvm-svn: 62407	2009-01-17 06:22:33 +00:00
Chris Lattner	f6a3bda65e	make the -rewrite-test a bit more interesting: it now ... wraps comments in <i> tags. Extend rewrite tokens to support this minimal functionality. llvm-svn: 57409	2008-10-12 06:09:52 +00:00
Chris Lattner	b6aa53b7de	start implementing a token rewriter. At this point, it just reads in a file ... and lets a client iterate over it. llvm-svn: 57407	2008-10-12 05:44:03 +00:00