Monday, September 11, 2006


POE 0.37

This weekend saw the release of POE 0.37 by Rocco. It includes all the tests I've written over the summer, and, since it's been released, the tests have been run successfully on a variety of platforms. The tests also found a few bugs, and those and other bugs have also been fixed in this release.

(The original release announcement.)

Friday, August 25, 2006


Final Notes

I;ve been asked to make a final post summarizing my progress. I already covered some of this in my last post, but here I will run down everything in the Pugs porting HowTo that I've completed. Items quoted from the HowTo will be italicized.

- $array[idx] -> @array[idx]
Done. This was on of the first translations I did, it was a simple matetr of finding any references to a single item of an array and changing the appororiate sigil. It was slightly harder to handle this inside text, since the text parser can only decide if a variable is a scalar, array, or hash at the end, but with some work I put together a parser that can handle single array or hash elements within text.

- $hash{$key} -> %hash{$key}
Done. This was actually a lot like the above translation, just on hashes.

- $hash{word} -> %hash
Done. This was simple, just a matter of finding and constnant keys to hashes and changing the opener and closer.

- "@array" -> "@array[]"
- "%hash" -> "%hash{}"
Done. These required a slightly smart parser for text. At the same time single array or hash elements are processed, entire hash or array references are handeled by just sticking on the appropraote {} or [].

- scalar @foo -> +@foo # or @foo.elems
Done. Currently this only translates to +@array, it would be good to add a .elems option for when the translator is doing heavy object orientation.

- a ? b : c -> a ?? b !! c
Done. Changed operators are easy to handle, I just look for any operators, and if it's a ? or : I translate it.

- $x =~ s/.../.../g -> $x ~~ s:P5:g/.../.../
Done. The translator converts all =~ to ~~, and can optionally use the :Perl5 modifier or try to translate the internals. Modifers (i, g, etc) are moved to the front (:i, :g) as approriate. In any case, remods are dropped after an attempt to translate them, since they'll only cause problems in Perl 6.

- foreach my $foo (@baz) {...} -> for @baz -> $foo {...}
Done. This is one translation that I am worried about breaking, since from the AST point of view foreach loops can look a lot different. However, in my testing the current code does translate all foreachs.

- length("foo") -> "foo".chars
Done. Fairly easy to handle, and fairly widely tested. At first I wasn't sure about my translation of this, but after wide testing and some revisions, I think it's fairly robust and accurate. It also works with variables instead of constants (length($foo) -> $foo.chars).

- Regular expressions:
m/ a * b /x -> m/ a * b / # /x now default
(space) -> # or \c[SPACE], \x20, \o40,
# <' '>
[abc] -> <[abc]>
[^abc] -> <-[abc]>
(?:...) -> [...]
< > -> \< \>

\p{prop} ->
\P{prop} -> <-prop>

\Qstring\E -> <{ quotemeta 'string' }> # or <'literal string'>

\A -> ^
\z -> $
\Z -> \n?$

\n -> \c[LF] # specifically a linefeed
\r?\n -> \n # logical newline
[^\n] -> \N # not a logical newline

\a -> \c[BEL]
\c[ -> \e
\cX -> \c[^X]
[^\t] -> \T
[^\r] -> \R
[^\f] -> \F
[^\e] -> \E
[^\x1B] -> \X1B
[^\x{263a}] -> \X[263a]

\1 -> $1
// -> //
/a|/ -> /a|/

x{2} -> x**{2}
x{2,} -> x**{2...}
x{1,2} -> x**{1..2}
x{1,2}? -> x**{1..2}?

(?=foo) ->
(?!foo) ->
(?<=foo) ->

(?{...}) -> {...}
(??{...}) -> <{...}>

(?>\d*) \d*: # single atom no backtrack
(?>(\d*)) (\d*): # () is still single atom
(?>...) [...]: # multiple atoms need []

s/foo/bar()/e -> s/foo/{ bar() }/
m?foo? -> m:once/foo/
Done. Since from the point of view of the AST a regex is justends up being just text I wrote a text parser that runs over regexs and changes all of these, as well as captures. Some of the more external considerations (?foo? -> m:once/foo/) are not handeled by the parser but by more functions, since from the inside both cases look the same.

- split(m/;/, $foo) -> split(/;/, $foo)
Done. Easy enough to catch, though at first I got carried away and got //m too.

- split(' ', $foo) -> words($foo) or $foo.words or «$foo»
Done. However, at present it always translates to .words, options will be added for the other cases.

- Heredocs:
< qq:to/END/
<<"END" -> qq:to/END/
<<'END' -> q:to/END/
Done. Once I finally had heredoc parsing from the yaml file correctly, this became easy, essentially just text substitution in the right places.

- # Perl 5
require Exporter;
our @ISA = qw<>;
our @EXPORT = qw<>;
sub foo { ... }

# -> Perl 6
sub foo(...) is export { ... }
Mostly Done. At current references to @ISA and @EXPORT (and @EXPORT_OKAY) are not removed, but then again I don't think this is a syntactic problem (they just become normal arrays that are never really used). For clarity they should probably be removed, and I'll try to add this to my next update.

- open()
open my $fh, "<", $filename or die $!; -> my $fh = open($filename, :r) err die $!;

"<" -> :r
">" -> :w
">>" -> :a
'+<" -> :rw

mixing them does not work yet

open(FH, $filename) -> our $fh = open($filename)
my $fh = open($filename)
and possibly:
sub FH () { $fh }
macro FH () { q:code { $fh } }
but $fh generally preferred
From Google's point of view (with my deadline) this one is untouched, however I have since added some of the basics. It's not done by anyone's standard, but to people that don't care about the deadline it's at least started. This is another place where TMTOWTDI makes it harder to translate, since the flexability of Perl5's open makes for many cases to translate.

- print to file
print $fh "Hello\n"; -> print $fh: "Hello\n"; # adding the colon
-> $fh.print("Hello\n");
-> $fh.print: "Hello\n";
Done. Currently only the add a "," option is support (which is roughly the same as the ":" from the Howto, but more recent accordin to lwall). A .print option should be added from heavy object-orientation.

- close the file
close($fh); -> the same or $fh.close;
Done. The optional .close is added when the translator is told to do all object oriented options (the -Oo option).

-    reading from a file
Done. At present only .readline is done, but the other option should be supported with time.

In addition, some things no explicitly in the howto are covered. Depending on user preference, 'no strict' can be used or undeclraed variables can be declared.

Overall, many of the most pressing issues for Perl 5 -> Perl 6 translation are covered.
The translator produces useable code in most cases, which can be fed to Pugs. While my work is not over, I think this code will ber very useable for the Perl community, especially when it's been even more extensively tested (I have yet to test on *nix at this point, but Mac OS X (close to *nix) and Windows are fine, given some complications with getting ghc to compile code for OS X).

The thing that worries me the most at present is that the translator still has some problems parsing or running. These errors occur only rarely (~2% or less of test files fail), but this is still not good enough for production software.

I'm proud of my work, but I also think I could have done better. In particular, my Haskell skills at the beginning were a little weak (but I've gotten much better), mostly since my school uses LISP to teach functional program and I had yet to code at this scale in a functional language. I'm also of the opinion that Haskell is not the best language to use for this kind of task, but that's just my personal taste. However, it was the best language to use in this case since Perl 6 isn't quite
to the point of running large, complex pieces of code quickly (though it will surely get there), and writing a perl 5 to perl 6 translator in perl 5 is a little backwards (the idea being that somebody that only has perl 6 can get their perl 5 code translated).

I hope I've done a service for the Perl community, and I'll continue to be as active as a wonky school schedule will allow. Thanks to everybody that helped, especially #perl6 which is like a second brain for those times I make a stupid mistake. It's been a great summer, and I look forward to continuing this work. In fact, I think that's one of the greates parts of this kind of setup: just cause I'm not getting payed for it anymore doesn't mean I can't keep working.

Sunday, August 20, 2006


Wrapping Up

Well, my time for Summer of Code is up. My code isn't complete, which of course means that I'll keep working on it, but as far as Google is concerned, this is what I produced. First, a rundown of what my code does:

-My translator will take a yaml parse of a perl 5 program and at least partially translate it (unless you tell it not to translate). Some yaml files still cause problems that I haven't been able to pin down, but around 98% of yaml files parse without errors.

-The translations that are handeled:
This is, unfortunately, not everything needed to translate Perl 5 to Perl 6. However, it is a start (maybe even a good start).

I'm hoping that my progress is considerd good enough for Summer of Code work. I know that I haven't actually complteted my project, mostly because many things I thought were easy ended up hard, and only a few things I thought were hard ended up easy. While the translator isn't complete, I've still provided some good work to the Perl community, and I hope to continue my work, even if my project is not considered a success.

At the very least, I can say I'm proud of my work. I revised my work daily most of the time, and I've learned much more then I thought I would.

I've made every effort to make my code open to others. If anybody wants to join me hacking my translator, it should be well enough documented to make working on it a little easier.

From here, I'm planning to continue work on my translator. First priority is bug fixes and added features, but I'd also like to start work on an expanded version of , since working with this file has shown me how incomplete it is.

Now that my Summer of Code project is (technically) over, I want to thank the Perl community for all the help and support. #perl6 has been particularly useful, as well as everybody I've talked to in the community. Unfortunately, the reality of college means I won't be able to keep up my current pace of development, but I'm not going to leave this code alone either. Before this project I had never been a part of an Open Source project to this degree, and the Perl community made diving into everything, from svn to ghc to anything, easy and interesting.

I'm going to make an effort to make some revisions in the remaining time before the "due" time for my Summer of Code work, but this is essentially my final post. I'll update (likely tomorrow) with any final revisions and thoughts.

Saturday, August 19, 2006


Two Days to Go

With only a couple of days left, here's a quick update before I get back to coding:

Testing has been broadened, covering what should be a representative sample of Perl 5 code, with good results. As expected, the percentage covered dropped a little across more code, but well over 95% of code is still parsed without errors (the actual coverage is ~98%+). I'm trying my hardest to get the last bits covered.

Few new translations were added this week. I've been feeding some code to Pugs, and the biggest problem tends to be with pre-decleration of variables. I've added an option to add "no strict;"
which should fix code, but "no strict;" isn't supported by pugs yet. I've been working on the mechanics of adding declerations, but haven't made enough progress for it to be viable yet. So, long story short, there's no real fix yet for the decleration of variables in files that don't already use strict.

I think that, in the long run, "no strict" is a good option since the presumption of the parser is that the Perl 5 code is valid and safe. For now, I'm working on a translation option to add "my" or "our" as needed to declare variables and make it strict.

With appropriate declerations, most translated code (including my classic test file, works fine in Pugs.

With my last two days, I'm going to try bug fixes, more testing, and hopefully get a variable decleration translation in.

Tuesday, August 15, 2006


STM Progress Report (9)

Done since last report:

Friday, August 11, 2006


The end is near...

All prophecies of doom aside, this week has been fairly productive. I've got a few more translations taken care of , as well as bug fixes. Documentation has been updated to reflect some new functionality, including the now completed option to do minimal regex changes using the :Perl5 modifier.

The code now runs without errors over ~98% of perl5's t directory (the actual percent covered is something like 97.77777...). However, my main problem is the appearance of unknown nodes. Since all possible nodes are now explicitly named, I know that Unknown nodes are appearing due to certain functions being called in places they shouldn't (many of my functions use an unknown node as a non-terminal way of returning an error). My first task of next week is to get rid of this problem.

I'm getting ready to really work my butt off this final week. Starting Monday I'll be back at new Mexico Tech, which means better internet access and plenty of free time. Hopefully I can get some real quality work done in the next week, to shore up some points that are still weak.

I'll still be updating next week, and I plan to post the week after that with a final report of what I've accomplished. Still, at this point I'd like to thank the Perl community for the opportunity to get to do this. I've learned an amazing amount this summer, and I look forward to continuing to work on Perl (and Pugs, and Parrot, etc...). Immediately following the Summer of Code I plan to continue revising and improving my translator, and helping in other areas when I can.

Thanks again for this great opportunity. Now to code...

Monday, August 07, 2006


STM Progress Report (8)

Done since last report:

This page is powered by Blogger. Isn't yours?