Test for the syntax highlighter

The point of this post is to test the syntax highlighter.

>>> a = ['Some', 'test']
>>> ', '.join(a)
'Some, test'

Seems I managed to convince it to work.

Gedit colorization

So, here's a gedit text colorization snippet.

Start with an empty document and open the bottom pane, with the Python console activated (View|Bottom Pane, Edit|Preferences|Plugins).

Enter this code:

>>> v = window.get_active_view()
>>> b = v.get_buffer()
>>> t = b.create_tag("simple_tag")
>>> b.insert(b.get_start_iter(), "hello")
>>> b.apply_tag(t, b.get_start_iter(), b.get_end_iter())

Nothing happens, we need to change the colors associated with the tag. It looks like I need to install some kind of support for inserting source code in blog posts. Tomorrow I will try these suggestions.

>>> import gtk
>>> t.props.foreground_gdk = gtk.gdk.Color("#fa0")

Tada! The text is now colorized. Since I'm selfish and sleepy, I won't go into details about the functions that were used (this post is mainly a self-note of how to perform text colorization). If anyone reads this and you feel like you could use some advice, try:

>>> dir(b)

'dir'ring python objects is helpful at times. Still, it would be better if the gedit/gtksourceview guys provided more documentation.

Good night to myself.

Resurrection... again

So I tried working on this project last year... and I I failed. Or at list I didn't get what I wanted, just a better understanding of monads and with that the conclusion that I don't like working with them.

It seems the old synpl project is like a fungus, it never dies if left alone in dark corners with high humidity.

I'm going to try again. This time, no super PL/SQL editor, no C compiler... but a structured editor.

Why? I'm very impressed with the usability of org-mode in Emacs and I want that kind of functionality (moving around content as if it was a tree) in other editing tasks.

I guess this experiment will die an unspectacular death like the previous ones. Well, if that's what it takes to keep the project alive...

Enough rambling. Time to actually use this blog space for something practical (and maybe useful to others).

Since I am not completely crazy, I don't want to implement an editor from scratch. I also don't want to use emacs (somehow elisp seems hard to use for large projects). I will use... gedit.

Yes, the puny notepad like editor distributed with Gnome.

Actually, it turns out that it is a very nice editor. For my purposes, at least. Because it has all the features I want (stylable editor, an API - in Python and explorable with a REPL, no less).

So my learning tasks are now these:

1. Learn the Gedit API.
2. Write an editor wrapper lib for the functionality I need (so I can switch to IronPython and ScintillaNET if I want to port my little scripts to Windows).
3. Write a Scheme/Lisp parser (or use an existing one) and test the usability of the structured approach.
4. Keep going with a JavaScript parser and structured editor.
5. Extend the JavaScript parser to support QooxDoo and project files (discovered automatically from exploring the file system neighborhood of the current file).
   
6. ... hmmm, profit? (mentally, at least)

Step number 1 is going to take a few blogposts. I intend to write small tutorials on programming the building blocks of my structured editor with the Gedit API. A preliminary list of these is:

1. Get the cursor position.
2. Set the cursor position.
3. Get the current selection.
4. Set the current selection.
5. Change text.
6. Colorize text.
7. Handle keyboard shortcuts.

Now that I know what I'll do with my free time, it's time to go to bed.

Resurrection attempt...

Hmmm... It looks like I've abandoned this project for almost one year.

Since I'm trying to learn Haskell and functional languages are supposed to be really good at handling compiling/translating/transforming tasks, it may be a good idea to try to reimplement things in Haskell. This and reading 'Real world Haskell' should complement nicely.

Here's a first attempt: a simple lexer written in less than 100 lines of Haskell (though it's pretty 'high density', without any comments).

A few tasks that should help me become a better Haskell programmer:

• document things (the literate Haskell mode works great with Emacs, no excuses this time); add examples of usage in the comments !
  
• learn how to isolate the lexer guts by using the Haskell module system
• use the State monad, because there is state passed around in the lexer (although in a rather uniform manner, which avoids 'ladders to the right of the screen')
• maybe use the Reader monad (there are some configuration items, such as the width of a tab)
• ... so maybe learn how to use monad transformers to 'stack' State and Reader?

Another change is that I decided that synpl needs to be open source this time. No foolish dreams of easy money :-) Anyway, since there is no way to submit archives to blogger, I should make a SourceForge or Google code project for this. Or, given that there are so many Haskell 'bosses' involved with Microsoft, maybe CodePlex...

Until then, here's my newbie Haskeller code (uncommented :-( ):

> data StreamPosition = StreamPosition { line :: Int, column :: Int }
>                       deriving Show

> data TokenPosition = TokenPosition { start :: StreamPosition,
>                                      end :: StreamPosition }
>                      deriving Show

> data TokenContent = TokenContent { position :: TokenPosition,
>                                    content :: String }
>                     deriving Show

> data Token = TokenNumber TokenContent
>            | TokenIdentifier TokenContent
>            | TokenChar TokenContent
>              deriving Show
> data LexerState = LexerState StreamPosition String [Token]

> lex :: String -> [Token]
> lex input = lexImpl $ LexerState (StreamPosition 1 1) input []
>     where lexImpl (LexerState pos [] tokens) = reverse tokens
>           lexImpl arg@(LexerState pos input@(c:cs) tokens)
>                     | c == '\n' = lexImpl $ LexerState (addToLine pos 1) cs tokens
>                     | c == ' ' = lexImpl $ LexerState (addToColumn pos 1) cs tokens
>                     | c == '\n' = lexImpl $ LexerState (addToColumn pos 8) cs tokens
>                     | isDigit c = lexImpl $ lexNumber arg
>                     | isLetterOrUnderscore c = lexImpl $ lexIdentifier arg
>                     | otherwise = lexImpl $ LexerState (addToColumn pos 1) cs upTokens
>                                 where upTokens = (charTok : tokens)
>                                       tokenPos = TokenPosition pos pos
>                                       charTok = TokenChar $ TokenContent tokenPos [c]
>           isDigit c = c >= '0' && c <= '9' 
>           isLetterOrUnderscore c = c == '_' || isLower || isUpper
>               where isLower = c >= 'a' && c <= 'z' 
>                     isUpper = c >= 'A' && c <= 'Z' 
>           lexNumber :: LexerState -> LexerState
>           lexNumber = lexEntity isDigit TokenNumber
>           lexIdentifier = lexEntity isLetterOrUnderscore TokenIdentifier
>           lexEntity :: (Char -> Bool) -> (TokenContent -> Token) -> LexerState -> LexerState
>           lexEntity spanP tokenCtor (LexerState pos cs tokens) = LexerState upPos upCs upTokens
>               where (content, upCs) = span spanP cs
>                     upPos = addToColumn pos (length content)
>                     tokenPos = TokenPosition pos (addToColumn upPos (-1))
>                     numTok = tokenCtor (TokenContent tokenPos content)
>                     upTokens = (numTok : tokens)
>           addToLine (StreamPosition line column) lineInc = StreamPosition (line + lineInc) column
>           addToColumn (StreamPosition line column) columnInc =
>               StreamPosition line (column + columnInc)

> data TokenTransform = TokenTransform { predicate :: ([Token] -> Int),
>                                        combiner :: ([Token] -> [Token]) }

> applyTokenTransform :: [Token] -> TokenTransform -> [Token]
> applyTokenTransform [] _ = []
> applyTokenTransform tokens transform@(TokenTransform p c) =
>     let matchLen = p tokens
>     in if matchLen > 0
>        then let (theMatch, theRest) = splitAt matchLen tokens
>                 theRest' = applyTokenTransform theRest transform
>             in (c theMatch) ++ theRest'
>        else let theHead = head tokens
>                 theRest = tail tokens
>             in (theHead : applyTokenTransform theRest transform)

> applyTokenTransforms :: [Token] -> [TokenTransform] -> [Token]
> applyTokenTransforms tokens (t:ts) = applyTokenTransforms (applyTokenTransform tokens t) ts
> applyTokenTransforms tokens [] = tokens

> matchPair :: Char -> Char -> TokenTransform
> matchPair char1 char2 = TokenTransform pred combiner
>     where isChar (TokenChar tc) ch = content tc == [ch]
>           isChar _ _ = False
>           pred [] = 0
>           pred [x] = 0
>           pred (el1 : el2 : els) | isChar el1 char1 && isChar el2 char2 = 2
>                                  | otherwise = 0
>           combiner (el1 : el2 : els) = (combineTokenChars el1 el2 : els)
>               where combineTokenChars (TokenChar tc1) (TokenChar tc2) =
>                         let startPos = (start . position)
>                             endPos = (end . position)
>                             newTokenPosition = TokenPosition (startPos tc1) (endPos tc2)
>                         in TokenChar $ TokenContent newTokenPosition (content tc1 ++ content tc2)

> matchOps :: [TokenTransform]
> matchOps = [matchPair '=' '=', matchPair '>' '=', matchPair '<' '=']  

Pretty printing comments

(Note: there's a new demo implementing what's described below; sadly, snapdrive.net seems to have a lot of downtime, if the link doesn't work please try later)

Pretty printing is simple, right? Just render the syntax tree as text in a pretty way and that's it.

Hmmm. What about comments? Some pretty printers (Synpl earlier versions included) just throw away comments, since they don't appear in the syntax tree. This is very wrong, as it loses precious information.

If we pretty print comments, where do we place them? The shape of the program may have changed drastically by reformatting it.

The solution I adopted is pretty naive, but it works rather well. This is what I do:

• when tokenizing the source, collect comments in a separate list; record the content and the starting and ending positions for each comment;
• allow some syntax tree nodes to have "after" and "before" comments; a good list of nodes to allow comments is: statements, item declarations in "DECLARE" sections of blocks, functions, procedures, triggers, packages (both spec and body) and types (also both spec and body);
• associate each comment with the closest node that accepts comments and doesn't overlap the comment;
  
• when pretty printing the node, print the comments with the same level of indentation as the node (for both "above" and "before" list of comments);

This works well, but... what about cases like:

begin
if 1 < a then
 c := 1;
else
 null;
 -- uncomment next PL/SQL line to remove warnings about variable c
 -- sometimes being used before initialization

 --c := 2;
end if;
null;
end;

There are three comments here. They should all be associated to the null statement after the else. This is what happens for the first two comments. But the third is "closer" to the second null statement, and is therefore inserted in its "before" list and shows between end if and null in the pretty print. This is clearly wrong. Also, the blank line between comments is lost in translation.

The solution? Group consecutive comments together into a larger comment, and preserve the blank lines in between.

After implementing comment grouping things work as expected. There are plenty of things left to chance, such as:

• what to do with code inside comments
• what about broken code inside comments
• what about formatting paragraphs in comments so they fit in the required number of columns?

Turns out pretty printing is an art in itself :)

New Synpl demo

Here's the first demo since switching languages to Scala.

What's in there:

• a parser for many SQL and PL/SQL language elements (blocks, basic variable definitions, most types of statements, most DML stuff, procedures, functions, packages)
  
• a basic analyzer (can look at a PL/SQL block and report variables that are used before being initialized, shadowed variables (same names used in inner blocks), variables that are defined and/or initialized but are never used (usage is not detected in all cases, and initialization by INTO clauses is not recognized)
  
• a pretty printer (adds missing elements - for instance, if the user forgot a 'THEN', the parser will report an error but recover and the pretty printer will show the source with all such errors corrected)
• a small GUI to help with testing
  

What's missing:

• many SQL and PL/SQL language elements are not identified by the parser (explicit join syntax for SQL, variable initialization at definition, PL/SQL objects and arrays etc.)
• most of the analyzer features
• advanced pretty printing (such as: detect total length of a SELECT query, and if it doesn't overflow the current line, print it on a single line)
  

Use 'run.bat' to launch the GUI. Use "File|Open" within the GUI to reach a sample PL/SQL source that demonstrates the error-recovery features of the parser, the current analyzer features and the pretty printer.


The GUI was tested with Java 1.6.

Synpl in Scala

I've decided to rewrite synpl in Scala. It's much better than C# for this particular task because of several reasons:

• it compiles to Java bytecodes, which means I can build a JAR file and then use it to build JDeveloper or SqlDeveloper extensions
• Scala has pattern matching and "case classes" which make it very easy to build AST trees and to walk those trees to do static analysis

So far, so good. These blog entries (1) (2) proved very useful as a Scala cheat sheet and saved me a lot of time.

Right now a significant chunk of the PL/SQL grammar is implemented and some static analysis also works (detecting variable use before initialization). There's plenty of work to be done, but it looks like Scala is a great language choice.

Probably tomorrow or the day after I'll put together a new demo.