Recognising strings and/or comments The Problem Simple patterns like: "/*"(.|\n)*"*/" {/*a C comment*/} (i.e. /*, followed by any characters, followed by */ ) will match everything in the input file from the first "/*" to the last "*/", including uncommented text, because lex always tries to match as much as it can with each pattern. We can overcome this if a single character indicates the end, as long as we don't allow nested comments - e.g. (i.e. "{", followed by any characters except a "}", terminated by "}" ) This should mean that recognising strings is easy, as they are usually terminated by the single character ". "\n" a newline "\"" a quote character "\\" a backslash character Note: this assumes that you only want to match correct input, and reject anything even slightly wrong. It is usually best to have two sets of patterns: the first being an exact match, and the second being more generous but always generating an error message, as it will only be used if the first version fails. The Solutions Either: Or:
My experiences with ANTLR Juten Tach, i wanted to summarize my first experiences with ANTLR a little bit. I have just recently finished my first little project for generating stub code for languages like Java or Actionscript. It was only a little project, but it had some interesting bits in it, that made it a good exercise. By the way, the whole sourcecode is available there, including the grammar. Everything’s running under the BSD license, so feel free to have a look at it.
Theory and Practice of Source Code Parsing with ANTLR and Roslyn provides several approaches to analysis of the source code written in different programming languages: This series of articles focuses on the structure and operation principles of the signature analysis module (PM, pattern matching). The key benefits of such an analyzer include high performance, simplicity of pattern description, and scalability across various languages. The disadvantage of this approach is that the module is not able to analyze complex vulnerabilities, which require developing high-level models of code execution. This article focuses on the first stage that includes parsing, comparing functionalities and features of various parsers, as well as applying theoretical principles to practice using Java, PHP, PLSQL, TSQL and even C# grammars. Thus far, in fact all the syntax of modern programming languages can be defined by context-free grammars. Moreover, the language may be context-free in one case and context-sensitive in the other. expr : expr '*' expr | expr '+' expr | or
Comp.compilers: Lexing nested comments From comp.compilers | List of all articles for this month | I'm working on a lexer for a language that has nested comments ("(*"&"*)") and nested pragmas ("<*" & "*>"). On the other hand, I've noticed none of gcc's frontends use a flex based lexer. David Starner - dstarner98@aasaa.ofe.org web, ftp: x8b4e53cd.dhcp.okstate.edu [It's easy to get lex to match nested comments, you just can't do it solely with regular expressions. %x COMMENT int nesting = 0; %% ... "(*" { BEGIN COMMENT; nesting = 0; /* initialized saved string */ } <COMMENT>"(*" { /* add to saved string for parser */ nesting++; } <COMMENT>"*)" { if(--nesting <=0 ) { BEGIN INITIAL; /* hand string to parser */ } /* add to saved string for parser */ } <COMMENT>. -John] Post a followup to this message Return to the comp.compilers page. Search the comp.compilers archives again.
JavaScript Target for the ANTLR Older revisions r125 by jhurstus on Aug 13, 2008 Diff creating 3.1 integration branch r93 by jhurstus on Aug 2, 2008 Diff merging /branches/upstream- integration/3.1b2 to trunk r79 by jhurstus on Jul 16, 2008 Diff merging branches/upstream- integration/3.1b1 through r78 to trunk All revisions of this file .NET Compiler Platform (a.k.a Roslyn) - An Overview Roslyn has been known as the code name for the next generation of C# compiler, at least since its first public preview was released in 2011. Infact the project started internally at Microsoft a couple of years earlier. Even before its first final release in Visual Studio 2015, it started to mean a lot more than just a new compiler. At that time, it also got a new official name: .NET Compiler Platform. Nevertheless, the word Roslyn is still a part of developer vocabularies and will probably remain in use for quite some time. Let us look at what one might be referring to today when mentioning Roslyn, what the current state of the project is, and how it can be beneficial to developers. This article is published from the DNC Magazine for .NET Developers and Architects. .NET Compiler Platform (a.k.a Roslyn) Most developers treat compilers as black boxes: they receive source code as input, do some processing on it, and output executable binaries. Image 1: Compiler as a black box Conclusion:
Justin Rogers : Language parsing and compiler design doesn't have to be hard, but boy this book really sucks! How'd you like that for an opening title? Did it grab your attention? Hell, your reading this far so I guess it did. The book I'm focusing on here is Build Your Own .NET Language and Compiler and please, don't click the link and then go buy it. The book starts out with the basics of parsing and regular expressions and all that jazz. OK, so you get to see a bunch of tools, and what do you get? OK, so forget the tools. At this point I want to identify the worst problem I found throughout the entire book. At the end of the book, it is apparent I'm not going to get anything of use and then it starts talking about code generation. How fair of a review is this? Lexer/Parser/Compiler Code and articles for different types of parsersLexer, Parser, Compiler, Oh My!
antlr: is there a simple example