synpl

Mono 2.4 Adventures

2009-07-04T15:02:00.000-07:00

I ended up writing a shell script (mostly copy pasting from this blogpost containing building instructions) that wgets and compiles Mono 2.4 including MonoDevelop.

I added it to the github repository (tools/get-and-build-mono.sh).

Turns out that using checkinstall instead of the classic 'make install' is a big mistake, as it creates debs and exposes my shiny new Mono 2.4 to the whims of 'apt', which promptly overwrites it on the first update. I should probably learn more about the Debian package system and apt, but right now simply putting everything in /opt/mono-2.4 and recompiling when needed (the script hopefully can now run unattended) works well enough for me.

Better way to install Mono 2.4

2009-07-04T10:07:00.000-07:00

Found at this address. Advantages: installs alongside "official" Mono.

Building Mono and MonoDevelop from source on Ubuntu 9.04 64bit

2009-07-02T10:27:00.001-07:00

Ok, I am officially fed up with the fact that the default MonoDevelop install on Ubuntu 9.04 isn't really able to do debugging.

Following this blog I found information on how to build Mono from sources. Things worked, with one minor glitch - I tried using 'make -j2' to use both cores, but compilation failed. Using only one core worked fine.

I also used checkinstall to make sure that 1) I can uninstall everything I compile cleanly and 2) I have .debs to reinstall later quickly if needed.

Now I need to compile Gtk#. Prerequisites:

sudo apt-get install libpango1.0-dev

sudo apt-get install libatk1.0-dev

sudo apt-get install libgtk2.0-dev

sudo apt-get install libglade2-dev

sudo apt-get install libgnomeui-dev libgnomecanvas2-dev libgnomeprint2.2-dev libgnomeprintui2.2-dev libpanel-applet2-dev libgnome2-dev

Compile and install gtk# (from http://ftp.novell.com/pub/mono/sources/gtk-sharp212/gtk-sharp-2.12.8.tar.bz2 for instance).

mono-addins (from http://ftp.novell.com/pub/mono/sources/mono-addins/mono-addins-0.4.tar.bz2)

gnome-sharp (http://ftp.novell.com/pub/mono/sources/gnome-sharp220/gnome-sharp-2.20.1.tar.bz2)

mono-tools (http://ftp.novell.com/pub/mono/sources/mono-tools/mono-tools-2.4.tar.bz2)

sudo apt-get install libncurses5-dev

mono-debugger (http://ftp.novell.com/pub/mono/sources/mono-debugger/mono-debugger-2.4.tar.bz2)

monodevelop (http://ftp.novell.com/pub/mono/sources/monodevelop/monodevelop-2.0.tar.bz2)

sudo apt-get install mozilla-dev (not sure this is actually required, it solves an error on my system).

monodevelop-debugger-mdb (http://ftp.novell.com/pub/mono/sources/monodevelop-debugger-mdb/monodevelop-debugger-mdb-2.0.tar.bz2)

nunit (from http://sourceforge.net/projects/nunit/files/NUnit%20Version%202/NUnit-2.4.8-
src.zip?download)

I ended up compiling nunit from source, using nunit-console.exe, nunit.core.dll and nunit.framework.dll as references in the test project and running nunit-console using something like:

mono ./nunit-console.exe Synpl.Test.Core.dll

to run my tests.

Using C# and Mono

2009-06-29T22:36:00.001-07:00

It turns out that using C# was a good decision. I was a bit disappointed with Monodevelop because it appeared to lack debugging support and refactoring, plus it has some issues when encapsulating private fields as properties. The missing debugger turned out to be a missing plugin, which I installed. Still, it feels a bit unfinished (crashes, apparent lockups etc.)

Other than these minor nuisances, the language feels great. API exploration is much easier with autocompletion. Monodevelop's Stetic GUI designer helps a lot with building the GUI without remembering much about Gtk#/Gtk+. It's true it is also buggy (I had to edit its XML file by hand to remove some cruft it added), but it is a timesaver nevertheless.

I did make some progress with the Sexp parser - I am now capable of editing (slowly) little LISP programs. I do need to add keyboard shortcut support (all the 'structured' editing is menu driven right now), probably also through IAbstractEditor so the platform dependent stuff stays isolated.

No regrets over switching to C#. Looking forward to further development of Synpl (I'm thinking that once I get the basic parts right with the Sexp parser, I should add a more sophisticated parser - something like a parser for JavaScript/Pascal/Basic to check the usability of 'structured' editing further).

Another Twist in the Road

2009-06-14T23:01:00.000-07:00

This weekend I thought about using C# for Synpl. This was the original plan when I started working on the "structured editor" variation of Synpl, but I gave up because I didn't want to tie it to Windows.

Since wxWidgets is such a great library, I tried to find a .NET wrapper for it. There is one, but it's not updated since 2007, and it doesn't work out of the box. Maybe the fact that I tried it on 64bit Ubuntu didn't help, but I had to abandon it anyway.

So I tried Mono and Monodevelop and Gtk#. Gtk again. This time, I managed to tweak the GtkTextView and associated classes to do what I want (for instance, raise events similar to Scintilla's Modification event). There were some bugs, but I managed to find some quick workarounds (for instance, the C# Gtk.TextBuffer class doesn't have a CreateTag method :-) ).

I made some very quick progress and I like the result a lot. Besides, working with autocompletion is so much better than trying to figure out things in the plain Python REPL or the Gedit Python REPL. I guess autocompletion in the editor is a very good complement to a REPL sometimes.

So I'm back to Gtk (well, Gtk#, but only the language is different), but not Gedit since there is no Gedit-sharp.

It did feel like "lateral progress", but the development speed is better. And the final product will be faster.

I need to port the Python code to C# ... good thing there isn't so much of it. I wonder if I'll regret switching from Python to C#.

Editor components API

2009-06-03T11:14:00.000-07:00

Today I managed to finish the tests for the "Text with changes" data structures described in a previous post.

I felt ready to create a basic Gedit plugin - but to my surprise, I found out that the API offered by Gedit doesn't offer a few essential notifications. I want to be able to register a callback with events such as 'buffer changes', regardless of cause (insertions, deletions, cuts, pastes, undos, redos etc.)

It turns out that the concepts used in the Gedit API are described in the PyGTK tutorial, but the event system in Gedit is even weaker than that in the TextView component of GTK (which doesn't offer generic enough 'buffer modification' events either).

I can hook up the keyboard events and handle insertions and deletions, but when it comes to cuts/pastes/undos/redos I have a hard time detecting the events (the user may use non-standard shortcuts) but, most importantly, I cannot isolate the text changes (which characters changed).

After a little more digging, I found out that 1) Scintilla supports a 'buffer modification' event that offers everything I need and 2) wxPython has a portable (GTK+/Win32) wrapper over Scintilla, along with demos with full source code that show how to use the control AND the 'buffer modification' notification/event.

So Synpl changes once again, not so radically this time. It was supposed to become a Gedit plugin, now it seems I'll end up with a standalone editor based on Scintilla and wxPython.

At least I feel like I'm making progress.

Synpl broken code uploaded on GitHub

2009-05-30T09:35:00.000-07:00

Yes, I know, broken code is not very useful to other people. I just wanted to have some kind of backup :-)

Today I worked pretty hard at finding a way to store text changes between the last successful parse and the current moment. Turns out that storing information about white space and comments in the parsed information is not very useful. I've given up on moving comments along with the item they're referring to. For now.

The most interesting result of today is that a very neat way of storing the text changes is a kind of a diff from the last successful parse. This can also be stored on disk very efficiently (before that I was thinking about storing the parse tree representation, which is much larger).

The only file that works right now is TextWithChanges.py in the root of the repository. The unit tests show what it can do. I'm pretty happy with the abstraction (I'm certain I'm rediscovering something classic here, just not realizing it clearly yet).

Basically, a TextWithChanges (TWC) stores the characters that were added or deleted since the last successful (local) parse. It can provide the 'old' version (that parses) and the current version, using current coordinates (which are actual cursor positions from the editor).

An old version of a slice of a TWC can look quite weird, as deleted characters share the same position (in terms of actual cursor positions).

"An example looks like this."

The red characters were deleted, the green ones added. The old version for a slice will not include the green characters, the current version will not include the red ones. Once a slice parses successfully, the changes are forgotten (the characters become black again).

Storing the version with changes on disk allows the editor to have access to a previous valid parse. Since it's possible to store only the changes and refer the file content with a hash, this can be very space-effective.

Clipboard managers for Gnome

2009-05-29T22:05:00.000-07:00

On Windows I use clipx to be more efficient when copying and pasting (no, it's not always code).

I'm playing again with Ubuntu and I miss clipx. Fortunately, there is parcellite which is a worthy replacement (Ubuntu packages).

With a bit of tweaking, I got it to behave almost like clipx. Copying and pasting is comfortable again!

Sexp structural editing

2009-05-27T22:58:00.000-07:00

Yesterday I managed to write some code should be very useful in creating a structured sexp editor plugin for Gedit. Need to clean it up a little bit, then post it to GitHub.

It's interesting how parsing for an editor needs to take into account detailed position information, whitespace and comments information (one of the tasks I want to accomplish is switching items in a sexp list without disturbing the existing layout - if possible - and move comments attached to an item along with the item). I previously thought that moving text chunks like that would require a very smart pretty printer, it turns out that may not be required.

However, there are still problems with languages with a more complicated syntax. I need to implement a parser for a simplified Basic to test the pretty printer-less approach in a more realistic setting.

Structured editing of sexp has long been accomplished in Emacs and lisp modes. I think it would be really interesting to see that extended to other languages, based on a lisp-like parse tree of the code.

Useful trick for interactive Gedit Python API exploration

2009-05-27T22:54:00.000-07:00

I mentioned the usefulness of 'dir' in trying to find out how to perform different tasks using the Gedit API. I should have thought about this earlier, but here is a dir+grep function that should help with the large number of methods associated with most Gedit API objects.


>>> def dir2(o, str):
...  return [k for k in dir(o) if k.find(str) != -1]
... 
>>> v = window.get_active_view()
>>> dir2(v, "selection")

This is very neat - it was stupid of me to browse those long method name lists without some automated help, especially since it was so easy to implement.

Final basic tasks for Gedit plugin API research

2009-05-25T12:38:00.000-07:00

I need to:

be able to add menus and keyboard shortcuts
be able to use dialogs and auto-completion like drop-downs
write a EAL (editor abstraction layer) so the Python code I write can be ported with reasonable ease to other editors that support Python scripting/modules

For task number 1, I will look into:

For task number 2:

Task number 3 needs more thought, hopefully the design will be somewhat clearer once tasks 1 and 2 are reasonably well researched and understood.

Gedit API - how to replace selected text

2009-05-25T12:05:00.000-07:00

Today's post is rather short. It is also a bit incomplete, as I don't understand one of the functions I use (even though the others seem very simple and obvious).


>>> v = window.get_active_view()
>>> b = v.get_buffer()
>>> b.insert_at_cursor("This is some sample text.\nHello, world!\n")

This is basic code to populate the editing buffer with some text to experiment on.


>>> sel0 = b.get_iter_at_offset(8)
>>> sel1 = b.get_iter_at_offset(12)
>>> b.select_range(sel0, sel1)

Selects the word "some" on the first line.


>>> b.delete_selection(0, 1)
>>> b.insert_at_cursor("emos")

The first line deletes the selected text. I don't know the meaning of the parameters - I tried several combinations and they delete the entire selection or seem to have no effect. So I just use (0, 1) as parameters without understanding.

The insert_at_cursor() call is already known and it inserts the string provided as parameter... at the current cursor position, obviously. Which is right where the selected text used to be, because we didn't move it.

Gedit API Text Coordinates

2009-05-20T22:30:00.001-07:00

I need to be able to convert between positions expressed as (line, column) pairs or (offset-in-characters-from-the-beginning-of-the-file). I also need to be able to get and set the cursor position and get and set the current selection. Also get the bounds of the document (first and last position).

The content of this post (and its "sister posts" about the Gedit API) is in a "exploratory notes" format. It doesn't provide a full API description, it is just meant to help me rediscover quickly how to perform certain tasks. I expected I would get this information from some blog somewhere, but it doesn't seem to be the case. I'm making these notes public because I suspect someone may find them useful at some point. If you're that someone, don't forget to use dir(object) in the Python console to explore further, and to browse the C APIs for Gtk/Gdk and Gedit if you need to dig deeper. I used to have some experience with Gtk, but that is a long time ago and I used it from C, not Python. So I'm a kind of newbie to Gtk/Gedit programming. I never used the Gnome APIs ;-)

Gedit uses the concepts of iterators and ranges to define positions in text and regions of text. An iterator can be used to inspect the text it points to, it can be moved forward and backward, it can be inspected to see if it is placed on the end of a line or word (this last feature is not that useful to me, as I'll have my own definitions for word/paragraph/block).

How do we get an iterator? Easy:


>>> v = window.get_active_view()
>>> b = v.get_buffer()
>>> b.insert_at_cursor("First line.\nSecond line\n.")
>>> start, end = b.get_bounds()
>>> end.get_line()
2
>>> end.get_line_offset()
0
>>> end.get_offset()
25

So, get_bounds() returns a sequence containing the iterator that points to the beginning of the document and the end of the document (the first two lines are pretty self explanatory, a "buffer" is a "document", the model displayed in the view).

"end" has a more interesting position ("start" is on offset 0, line 0, column 0, obviously). get_line() returns the line (counting from 0 as the first line), get_line_offset() returns the column (also counting from 0 as the first column) and get_offset() returns the number of characters since the beginning of the file (I suspect this will be the information I will use, as it is easily converted to line/column formats).

Next: look at the text pointed by an iterator, move the iterator.


>>> start.get_chars_in_line()
12
>>> start.get_char()
u'F'
>>> start.forward_char()
True
>>> start.get_char()
u'i'
>>> start.forward_to_line_end()
True
>>> start.get_char()
u'\n'
>>> start.get_offset()
11
>>> start.get_line()
0
>>> start.get_line_offset()
11
>>> start.forward_char()
True
>>> start.get_offset()
12
>>> start.get_line()
1
>>> start.get_line_offset()
0

So I've looked at the char pointed to by the iterator, moved the iterator and played with its position to make sure my assumptions about get_line(), get_line_offset() and get_offset() are correct.

It's a bit weird to use an iterator named "start" to do all this, but since iterators are mutable objects, it worked. BTW, we didn't affect the beginning of the file in any way. "start" just happened to begin its life by pointing at the beginning of the file, that's it.

How to set the position of an iterator if we know the "offset", for instance?


>>> start.get_offset()
12
>>> end.set_offset(start.get_offset())
>>> end.get_offset()
12
>>> end.get_char()
u'S'
>>> start.get_char()
u'S'
>>> end.get_line()
1
>>> end.get_line_offset()
0

So end and start point to the same position, and I can use an iterator to convert between the offset-from-BOF and (line, column) representations of a position. I can also set the line and column via set_line() and set_line_offset() then get the offset-from-BOF using get_offset(), so the conversion works both ways.

How do I get the iterator at cursor? Place the cursor at the end of the word "Second" on the second line of text using the mouse.


>>> b.get_insert()

>>> b.get_iter_at_mark(b.get_insert())

>>> at_cursor = b.get_iter_at_mark(b.get_insert())
>>> at_cursor.get_line()
1
>>> at_cursor.get_line_offset()
6
>>> at_cursor.get_char()
u' '

I don't know what a mark is yet, and I don't know if these iterators and marks don't need to be removed somehow to avoid memory leaks. However, I'm not too worried about that now. If it becomes a problem, I'll fix it.

The "at_cursor" iterator seems to be positioned properly, though.

How to position the cursor using an iterator (the reverse of getting the position of the cursor as an iterator)? I'll just try to move the cursor over the space between "Second" and "line", then get the position of the cursor again and checked it moved.


>>> at_cursor.forward_char()
True
>>> b.place_cursor(at_cursor)
>>> at_cursor_new = b.get_iter_at_mark(b.get_insert())
>>> at_cursor_new.get_offset()
19
>>> at_cursor.get_offset()
19

So the relevant call is "b.place_cursor()".

What if I want to find out the coordinates of the current selection? Make sure no text is selected.


>>> b.get_selection_bounds()
()

Now select "ond" in "Second" on the second line.


>>> b.get_selection_bounds()
(, )
>>> sel0, sel1 = b.get_selection_bounds()
>>> sel0.get_offset()
15
>>> sel1.get_offset()
18
>>> sel1.get_char()
u' '

So "get_selection_bounds()" returns two iterators, one for the begining and the other for the end of the selection. "sel1" points to the first character after the selection.

How do I set the selection? Deselect the text.


>>> b.select_range(sel0, sel1)

The text is selected again.

How do I get the text between two iterators (using a loop and get_char() seems wasteful, especially since Python strings are immutable)?


>>> b.get_text(sel0, sel1)
'ond'

That concludes the things I wanted to explore in this post. Again, these are just a newbie's personal notes and I will use them to draft a plugin. There are probably better ways to accomplish the tasks described in this post. I was just happy to find one way to perform them, given the lack of documentation (this perception of the documentation is very subjective, some APIs have far worse documentation, some far better; however, since I use the Python "dir()" function and a lot of trial and error, I dare say "lack of documentation").

Test for the syntax highlighter

2009-05-18T22:16:00.001-07:00

The point of this post is to test the syntax highlighter.


>>> a = ['Some', 'test']
>>> ', '.join(a)
'Some, test'

Seems I managed to convince it to work.

Gedit colorization

2009-05-18T13:24:00.000-07:00

So, here's a gedit text colorization snippet.

Start with an empty document and open the bottom pane, with the Python console activated (View|Bottom Pane, Edit|Preferences|Plugins).

Enter this code:


>>> v = window.get_active_view()
>>> b = v.get_buffer()
>>> t = b.create_tag("simple_tag")
>>> b.insert(b.get_start_iter(), "hello")
>>> b.apply_tag(t, b.get_start_iter(), b.get_end_iter())

Nothing happens, we need to change the colors associated with the tag. It looks like I need to install some kind of support for inserting source code in blog posts. Tomorrow I will try these suggestions.


>>> import gtk
>>> t.props.foreground_gdk = gtk.gdk.Color("#fa0")

Tada! The text is now colorized. Since I'm selfish and sleepy, I won't go into details about the functions that were used (this post is mainly a self-note of how to perform text colorization). If anyone reads this and you feel like you could use some advice, try:


>>> dir(b)

'dir'ring python objects is helpful at times. Still, it would be better if the gedit/gtksourceview guys provided more documentation.

Good night to myself.

Resurrection... again

2009-05-18T13:10:00.000-07:00

So I tried working on this project last year... and I I failed. Or at list I didn't get what I wanted, just a better understanding of monads and with that the conclusion that I don't like working with them.

It seems the old synpl project is like a fungus, it never dies if left alone in dark corners with high humidity.

I'm going to try again. This time, no super PL/SQL editor, no C compiler... but a structured editor.

Why? I'm very impressed with the usability of org-mode in Emacs and I want that kind of functionality (moving around content as if it was a tree) in other editing tasks.

I guess this experiment will die an unspectacular death like the previous ones. Well, if that's what it takes to keep the project alive...

Enough rambling. Time to actually use this blog space for something practical (and maybe useful to others).

Since I am not completely crazy, I don't want to implement an editor from scratch. I also don't want to use emacs (somehow elisp seems hard to use for large projects). I will use... gedit.

Yes, the puny notepad like editor distributed with Gnome.

Actually, it turns out that it is a very nice editor. For my purposes, at least. Because it has all the features I want (stylable editor, an API - in Python and explorable with a REPL, no less).

So my learning tasks are now these:

Learn the Gedit API.
Write an editor wrapper lib for the functionality I need (so I can switch to IronPython and ScintillaNET if I want to port my little scripts to Windows).
Write a Scheme/Lisp parser (or use an existing one) and test the usability of the structured approach.
Keep going with a JavaScript parser and structured editor.
Extend the JavaScript parser to support QooxDoo and project files (discovered automatically from exploring the file system neighborhood of the current file).
... hmmm, profit? (mentally, at least)

Step number 1 is going to take a few blogposts. I intend to write small tutorials on programming the building blocks of my structured editor with the Gedit API. A preliminary list of these is:

Get the cursor position.
Set the cursor position.
Get the current selection.
Set the current selection.
Change text.
Colorize text.
Handle keyboard shortcuts.

Now that I know what I'll do with my free time, it's time to go to bed.

Resurrection attempt...

2008-07-16T13:17:00.000-07:00

Hmmm... It looks like I've abandoned this project for almost one year.

Since I'm trying to learn Haskell and functional languages are supposed to be really good at handling compiling/translating/transforming tasks, it may be a good idea to try to reimplement things in Haskell. This and reading 'Real world Haskell' should complement nicely.

Here's a first attempt: a simple lexer written in less than 100 lines of Haskell (though it's pretty 'high density', without any comments).

A few tasks that should help me become a better Haskell programmer:

document things (the literate Haskell mode works great with Emacs, no excuses this time); add examples of usage in the comments !
learn how to isolate the lexer guts by using the Haskell module system
use the State monad, because there is state passed around in the lexer (although in a rather uniform manner, which avoids 'ladders to the right of the screen')
maybe use the Reader monad (there are some configuration items, such as the width of a tab)
... so maybe learn how to use monad transformers to 'stack' State and Reader?

Another change is that I decided that synpl needs to be open source this time. No foolish dreams of easy money :-) Anyway, since there is no way to submit archives to blogger, I should make a SourceForge or Google code project for this. Or, given that there are so many Haskell 'bosses' involved with Microsoft, maybe CodePlex...

Until then, here's my newbie Haskeller code (uncommented :-( ):



> data StreamPosition = StreamPosition { line :: Int, column :: Int }
>                       deriving Show

> data TokenPosition = TokenPosition { start :: StreamPosition,
>                                      end :: StreamPosition }
>                      deriving Show

> data TokenContent = TokenContent { position :: TokenPosition,
>                                    content :: String }
>                     deriving Show

> data Token = TokenNumber TokenContent
>            | TokenIdentifier TokenContent
>            | TokenChar TokenContent
>              deriving Show
> data LexerState = LexerState StreamPosition String [Token]

> lex :: String -> [Token]
> lex input = lexImpl $ LexerState (StreamPosition 1 1) input []
>     where lexImpl (LexerState pos [] tokens) = reverse tokens
>           lexImpl arg@(LexerState pos input@(c:cs) tokens)
>                     | c == '\n' = lexImpl $ LexerState (addToLine pos 1) cs tokens
>                     | c == ' ' = lexImpl $ LexerState (addToColumn pos 1) cs tokens
>                     | c == '\n' = lexImpl $ LexerState (addToColumn pos 8) cs tokens
>                     | isDigit c = lexImpl $ lexNumber arg
>                     | isLetterOrUnderscore c = lexImpl $ lexIdentifier arg
>                     | otherwise = lexImpl $ LexerState (addToColumn pos 1) cs upTokens
>                                 where upTokens = (charTok : tokens)
>                                       tokenPos = TokenPosition pos pos
>                                       charTok = TokenChar $ TokenContent tokenPos [c]
>           isDigit c = c >= '0' && c <= '9' 
>           isLetterOrUnderscore c = c == '_' || isLower || isUpper
>               where isLower = c >= 'a' && c <= 'z' 
>                     isUpper = c >= 'A' && c <= 'Z' 
>           lexNumber :: LexerState -> LexerState
>           lexNumber = lexEntity isDigit TokenNumber
>           lexIdentifier = lexEntity isLetterOrUnderscore TokenIdentifier
>           lexEntity :: (Char -> Bool) -> (TokenContent -> Token) -> LexerState -> LexerState
>           lexEntity spanP tokenCtor (LexerState pos cs tokens) = LexerState upPos upCs upTokens
>               where (content, upCs) = span spanP cs
>                     upPos = addToColumn pos (length content)
>                     tokenPos = TokenPosition pos (addToColumn upPos (-1))
>                     numTok = tokenCtor (TokenContent tokenPos content)
>                     upTokens = (numTok : tokens)
>           addToLine (StreamPosition line column) lineInc = StreamPosition (line + lineInc) column
>           addToColumn (StreamPosition line column) columnInc =
>               StreamPosition line (column + columnInc)

> data TokenTransform = TokenTransform { predicate :: ([Token] -> Int),
>                                        combiner :: ([Token] -> [Token]) }

> applyTokenTransform :: [Token] -> TokenTransform -> [Token]
> applyTokenTransform [] _ = []
> applyTokenTransform tokens transform@(TokenTransform p c) =
>     let matchLen = p tokens
>     in if matchLen > 0
>        then let (theMatch, theRest) = splitAt matchLen tokens
>                 theRest' = applyTokenTransform theRest transform
>             in (c theMatch) ++ theRest'
>        else let theHead = head tokens
>                 theRest = tail tokens
>             in (theHead : applyTokenTransform theRest transform)

> applyTokenTransforms :: [Token] -> [TokenTransform] -> [Token]
> applyTokenTransforms tokens (t:ts) = applyTokenTransforms (applyTokenTransform tokens t) ts
> applyTokenTransforms tokens [] = tokens

> matchPair :: Char -> Char -> TokenTransform
> matchPair char1 char2 = TokenTransform pred combiner
>     where isChar (TokenChar tc) ch = content tc == [ch]
>           isChar _ _ = False
>           pred [] = 0
>           pred [x] = 0
>           pred (el1 : el2 : els) | isChar el1 char1 && isChar el2 char2 = 2
>                                  | otherwise = 0
>           combiner (el1 : el2 : els) = (combineTokenChars el1 el2 : els)
>               where combineTokenChars (TokenChar tc1) (TokenChar tc2) =
>                         let startPos = (start . position)
>                             endPos = (end . position)
>                             newTokenPosition = TokenPosition (startPos tc1) (endPos tc2)
>                         in TokenChar $ TokenContent newTokenPosition (content tc1 ++ content tc2)

> matchOps :: [TokenTransform]
> matchOps = [matchPair '=' '=', matchPair '>' '=', matchPair '<' '=']

Pretty printing comments

2007-08-31T00:21:00.000-07:00

(Note: there's a new demo implementing what's described below; sadly, snapdrive.net seems to have a lot of downtime, if the link doesn't work please try later)

Pretty printing is simple, right? Just render the syntax tree as text in a pretty way and that's it.

Hmmm. What about comments? Some pretty printers (Synpl earlier versions included) just throw away comments, since they don't appear in the syntax tree. This is very wrong, as it loses precious information.

If we pretty print comments, where do we place them? The shape of the program may have changed drastically by reformatting it.

The solution I adopted is pretty naive, but it works rather well. This is what I do:

when tokenizing the source, collect comments in a separate list; record the content and the starting and ending positions for each comment;
allow some syntax tree nodes to have "after" and "before" comments; a good list of nodes to allow comments is: statements, item declarations in "DECLARE" sections of blocks, functions, procedures, triggers, packages (both spec and body) and types (also both spec and body);
associate each comment with the closest node that accepts comments and doesn't overlap the comment;
when pretty printing the node, print the comments with the same level of indentation as the node (for both "above" and "before" list of comments);

This works well, but... what about cases like:


begin
if 1 < a then
 c := 1;
else
 null;
 -- uncomment next PL/SQL line to remove warnings about variable c
 -- sometimes being used before initialization

 --c := 2;
end if;
null;
end;

There are three comments here. They should all be associated to the null statement after the else. This is what happens for the first two comments. But the third is "closer" to the second null statement, and is therefore inserted in its "before" list and shows between end if and null in the pretty print. This is clearly wrong. Also, the blank line between comments is lost in translation.

The solution? Group consecutive comments together into a larger comment, and preserve the blank lines in between.

After implementing comment grouping things work as expected. There are plenty of things left to chance, such as:

what to do with code inside comments
what about broken code inside comments
what about formatting paragraphs in comments so they fit in the required number of columns?

Turns out pretty printing is an art in itself :)

New Synpl demo

2007-08-17T05:38:00.000-07:00

Here's the first demo since switching languages to Scala.

What's in there:

a parser for many SQL and PL/SQL language elements (blocks, basic variable definitions, most types of statements, most DML stuff, procedures, functions, packages)
a basic analyzer (can look at a PL/SQL block and report variables that are used before being initialized, shadowed variables (same names used in inner blocks), variables that are defined and/or initialized but are never used (usage is not detected in all cases, and initialization by INTO clauses is not recognized)
a pretty printer (adds missing elements - for instance, if the user forgot a 'THEN', the parser will report an error but recover and the pretty printer will show the source with all such errors corrected)
a small GUI to help with testing

What's missing:

many SQL and PL/SQL language elements are not identified by the parser (explicit join syntax for SQL, variable initialization at definition, PL/SQL objects and arrays etc.)
most of the analyzer features
advanced pretty printing (such as: detect total length of a SELECT query, and if it doesn't overflow the current line, print it on a single line)

Use 'run.bat' to launch the GUI. Use "File|Open" within the GUI to reach a sample PL/SQL source that demonstrates the error-recovery features of the parser, the current analyzer features and the pretty printer.

The GUI was tested with Java 1.6.

Synpl in Scala

2007-08-16T07:42:00.000-07:00

I've decided to rewrite synpl in Scala. It's much better than C# for this particular task because of several reasons:

it compiles to Java bytecodes, which means I can build a JAR file and then use it to build JDeveloper or SqlDeveloper extensions
Scala has pattern matching and "case classes" which make it very easy to build AST trees and to walk those trees to do static analysis

So far, so good. These blog entries (1) (2) proved very useful as a Scala cheat sheet and saved me a lot of time.

Right now a significant chunk of the PL/SQL grammar is implemented and some static analysis also works (detecting variable use before initialization). There's plenty of work to be done, but it looks like Scala is a great language choice.

Probably tomorrow or the day after I'll put together a new demo.

New Ideas

2007-06-27T15:50:00.000-07:00

I've been thinking about synpl and applications of a SQL and PL/SQL source code analyzer for some time now.

Here's some of the ideas that I had (some of them don't require a parser):

build a visual representation of the database; have the triggers/constraints listed next to each table - I understand Visual Studio offers something like this, even for Oracle, I need to look into this;
look into the code and find out what indexes should exist (analyze WHERE clauses and INSERTs, try to find a balance between them)
analyze where clauses and discover undeclared foreign keys
build a measure of the connection between tables (strongly connected vs. weakly connected) and then build sets of closely related tables - could help people trying to understand vast databases with lots of tables by pointing out which tables are most closely related; trying to understand a group of 10 tables that are mostly connected to one another and have a few connections to other tables is much easier than trying to understand a group of 50 tables;
build a Django admin-like app with Java hooks (load classes for validation, preprocessing, postprocessing, custom actions, reporting); Application Express does this already? need to look into it;
analyze INSERTs and UPDATEs and try to infer if some fields only have a limited number of values (missing restrictions?)

Parsing the SQL queries to discover connections between tables should go beyond SQL queries in PL/SQL (i.e. use SQL queries in other languages) but the source code isn't always available. It's very rarely available, in fact.

New synpl demo

2007-06-24T04:34:00.000-07:00

I've prepared a new demo of synpl.

It's probably broken in many places as I didn't get a chance to test it properly.

What's new:

the ability to 'correct' slightly incorrect syntax (such as missing a ';' or an 'if' in 'end if')
it can show the corrected version in a separate window; the corrections are highlighted
it can load/save samples

Plans

2007-06-19T01:10:00.000-07:00

I'm thinking of switching Synpl to Java, so that I can write a JDeveloper extension - and later a SqlDeveloper one. The problem is that the JDeveloper extension APIs aren't that well documented. There are even two competing APIs, one that is specific to Oracle and the other which is a JSR (198, to be more precise).

There are illuminating code samples for the Oracle API (which I hope can be used to develop an extension if combined with the API's JavaDoc). On the other hand, the JSR 198 seems more restrictive. One one forum discussion someone was complaining about the fact that it's impossible to write a code formatter with the JSR 198 API. I'm not sure that's true, but from what I've seen in the JavaDoc it's certainly a difficult task.

Hmmm. Back to the GIS application and paper ;)

Why?

2007-06-14T01:42:00.000-07:00

What's synpl? A simple PL/SQL parser (for now) and analyzer (in the near future) written in an attempt to provide friendly compile/parse error messages and warnings about problems that the compiler is silent about.

Since I need to show synpl to people and email seems to be a tedious way to do it, I've started this blog to help me communicate with users willing to test synpl and provide feedback.

The first available demo is here. You need Windows and .NET framework 2 to run it (the GUI seems to be running fine under Mono 1.2.4 on Debian, but it may break in the future).