Summer of Code 2006

POE 0.37

2006-09-11T01:57:00.000-07:00

This weekend saw the release of POE 0.37 by Rocco. It includes all the tests I've written over the summer, and, since it's been released, the tests have been run successfully on a variety of platforms. The tests also found a few bugs, and those and other bugs have also been fixed in this release.

(The original release announcement.)

Final Notes

2006-08-25T10:20:00.000-07:00

I;ve been asked to make a final post summarizing my progress. I already covered some of this in my last post, but here I will run down everything in the Pugs porting HowTo that I've completed. Items quoted from the HowTo will be italicized.

- $array[idx] -> @array[idx]

Done. This was on of the first translations I did, it was a simple matetr of finding any references to a single item of an array and changing the appororiate sigil. It was slightly harder to handle this inside text, since the text parser can only decide if a variable is a scalar, array, or hash at the end, but with some work I put together a parser that can handle single array or hash elements within text.

- $hash{$key} -> %hash{$key}

Done. This was actually a lot like the above translation, just on hashes.

- $hash{word} -> %hash

Done. This was simple, just a matter of finding and constnant keys to hashes and changing the opener and closer.

- "@array" -> "@array[]"
- "%hash" -> "%hash{}"

Done. These required a slightly smart parser for text. At the same time single array or hash elements are processed, entire hash or array references are handeled by just sticking on the appropraote {} or [].

- scalar @foo -> +@foo # or @foo.elems

Done. Currently this only translates to +@array, it would be good to add a .elems option for when the translator is doing heavy object orientation.

- a ? b : c -> a ?? b !! c

Done. Changed operators are easy to handle, I just look for any operators, and if it's a ? or : I translate it.

- $x =~ s/.../.../g -> $x ~~ s:P5:g/.../.../

Done. The translator converts all =~ to ~~, and can optionally use the :Perl5 modifier or try to translate the internals. Modifers (i, g, etc) are moved to the front (:i, :g) as approriate. In any case, remods are dropped after an attempt to translate them, since they'll only cause problems in Perl 6.

- foreach my $foo (@baz) {...} -> for @baz -> $foo {...}

Done. This is one translation that I am worried about breaking, since from the AST point of view foreach loops can look a lot different. However, in my testing the current code does translate all foreachs.

- length("foo") -> "foo".chars

Done. Fairly easy to handle, and fairly widely tested. At first I wasn't sure about my translation of this, but after wide testing and some revisions, I think it's fairly robust and accurate. It also works with variables instead of constants (length($foo) -> $foo.chars).

- Regular expressions:
    m/ a * b /x       -> m/ a * b /               # /x now default
    (space)           ->                      # or \c[SPACE], \x20, \o40,
                                                  # <' '>
    [abc]             -> <[abc]>
    [^abc]            -> <-[abc]>
    (?:...)           -> [...]
    <  >              -> \<  \>

    \p{prop}          -> 
    \P{prop}          -> <-prop>

    \Qstring\E        -> <{ quotemeta 'string' }> # or <'literal string'>

    \A                -> ^
    \z                -> $
    \Z                -> \n?$

    \n                -> \c[LF]                   # specifically a linefeed
    \r?\n             -> \n                       # logical newline
    [^\n]             -> \N                       # not a logical newline

    \a                -> \c[BEL]
    \N{CENT SIGN}     -> \c[CENT SIGN]
    [^\N{CENT SIGN}]  -> \C[CENT SIGN]
    \c[               -> \e
    \cX               -> \c[^X]
    [^\t]             -> \T
    [^\r]             -> \R
    [^\f]             -> \F
    [^\e]             -> \E
    [^\x1B]           -> \X1B
    [^\x{263a}]       -> \X[263a]

    \1                -> $1
    //                -> //
    /a|/              -> /a|/

    x{2}              -> x**{2}
    x{2,}             -> x**{2...}
    x{1,2}            -> x**{1..2}
    x{1,2}?           -> x**{1..2}?

    (?=foo)           ->
    (?!foo)           -> 
    (?<=foo)          ->
    (? 

    (?{...})          -> {...}
    (??{...})         -> <{...}>

    (?>\d*)           \d*:                    # single atom no backtrack
    (?>(\d*))         (\d*):                  # () is still single atom
    (?>...)           [...]:                  # multiple atoms need []

    s/foo/bar()/e     -> s/foo/{ bar() }/
    m?foo?            -> m:once/foo/

Done. Since from the point of view of the AST a regex is justends up being just text I wrote a text parser that runs over regexs and changes all of these, as well as captures. Some of the more external considerations (?foo? -> m:once/foo/) are not handeled by the parser but by more functions, since from the inside both cases look the same.

- split(m/;/, $foo) -> split(/;/, $foo)

Done. Easy enough to catch, though at first I got carried away and got //m too.

- split(' ', $foo) -> words($foo) or $foo.words or «$foo»

Done. However, at present it always translates to .words, options will be added for the other cases.

- Heredocs:
    < qq:to/END/
    <<"END"  -> qq:to/END/
    <<'END'  ->  q:to/END/

Done. Once I finally had heredoc parsing from the yaml file correctly, this became easy, essentially just text substitution in the right places.

- # Perl 5
  require Exporter;
  our @ISA    = qw<>;
  our @EXPORT = qw<>;
  sub foo { ... }

  # -> Perl 6
  sub foo(...) is export { ... }

Mostly Done. At current references to @ISA and @EXPORT (and @EXPORT_OKAY) are not removed, but then again I don't think this is a syntactic problem (they just become normal arrays that are never really used). For clarity they should probably be removed, and I'll try to add this to my next update.

- open()
     open my $fh, "<", $filename or die $!;    ->   my $fh = open($filename, :r) err die $!;

     "<"   -> :r
     ">"   -> :w
     ">>"  -> :a
     '+<"  -> :rw

     mixing them does not work yet

     open(FH, $filename)     ->  our $fh = open($filename)
                                  my $fh = open($filename)
                              and possibly:
                                  sub FH () { $fh }
                                  macro FH () { q:code { $fh } }
                              but $fh generally preferred

From Google's point of view (with my deadline) this one is untouched, however I have since added some of the basics. It's not done by anyone's standard, but to people that don't care about the deadline it's at least started. This is another place where TMTOWTDI makes it harder to translate, since the flexability of Perl5's open makes for many cases to translate.

- print to file
     print $fh "Hello\n";    -> print $fh: "Hello\n";   # adding the colon
                                or
                             -> $fh.print("Hello\n");
                             -> $fh.print: "Hello\n";

Done. Currently only the add a "," option is support (which is roughly the same as the ":" from the Howto, but more recent accordin to lwall). A .print option should be added from heavy object-orientation.

- close the file
   close($fh);               -> the same or $fh.close;

Done. The optional .close is added when the translator is told to do all object oriented options (the -Oo option).

-    reading from a file
   $fh.readline
   =$fh

Done. At present only .readline is done, but the other option should be supported with time.

In addition, some things no explicitly in the howto are covered. Depending on user preference, 'no strict' can be used or undeclraed variables can be declared.

Overall, many of the most pressing issues for Perl 5 -> Perl 6 translation are covered.
The translator produces useable code in most cases, which can be fed to Pugs. While my work is not over, I think this code will ber very useable for the Perl community, especially when it's been even more extensively tested (I have yet to test on *nix at this point, but Mac OS X (close to *nix) and Windows are fine, given some complications with getting ghc to compile code for OS X).

The thing that worries me the most at present is that the translator still has some problems parsing or running. These errors occur only rarely (~2% or less of test files fail), but this is still not good enough for production software.

I'm proud of my work, but I also think I could have done better. In particular, my Haskell skills at the beginning were a little weak (but I've gotten much better), mostly since my school uses LISP to teach functional program and I had yet to code at this scale in a functional language. I'm also of the opinion that Haskell is not the best language to use for this kind of task, but that's just my personal taste. However, it was the best language to use in this case since Perl 6 isn't quite
to the point of running large, complex pieces of code quickly (though it will surely get there), and writing a perl 5 to perl 6 translator in perl 5 is a little backwards (the idea being that somebody that only has perl 6 can get their perl 5 code translated).

I hope I've done a service for the Perl community, and I'll continue to be as active as a wonky school schedule will allow. Thanks to everybody that helped, especially #perl6 which is like a second brain for those times I make a stupid mistake. It's been a great summer, and I look forward to continuing this work. In fact, I think that's one of the greates parts of this kind of setup: just cause I'm not getting payed for it anymore doesn't mean I can't keep working.

Wrapping Up

2006-08-20T21:02:00.000-07:00

Well, my time for Summer of Code is up. My code isn't complete, which of course means that I'll keep working on it, but as far as Google is concerned, this is what I produced. First, a rundown of what my code does:

-My translator will take a yaml parse of a perl 5 program and at least partially translate it (unless you tell it not to translate). Some yaml files still cause problems that I haven't been able to pin down, but around 98% of yaml files parse without errors.

-The translations that are handeled:

Sigils are translated. This handels issues such as $hash{key} -> %hash{key} (since sigils are now invariate). Sigils are handled both in use and imbedded in text (and the AST definition of text includes regexs).
Hash keys are adjusted, in particular for constant keys.
Conditional expressions are translated. In effect, every ? operator becomes ?? and every : operator becomes !!.
Regexs are now used with ~~. The =~ operator becomes ~~.
Regex modifiers, such as //g or //i, are moved to the beginning, m:g// and m:i// (or many at the same time).
The /e regex modifier is now default, so it's just dropped.
becomes a method call to .readline on the $fh object.
The m at the beginning of a regex in a split is removed.
length function calls become .chars method calls.
File handles are closed with a method call to .close
Splitting on a static space becomes a method call to .words
foreach loops have a new syntax, going from foreach $foo (@bar){} to foreach @bar -> $foo
Just about everything inside regexs translates. This includes metacharacters, counts, captures, backtracking, lookahead, and all that other (?...) stuff. This is the majority of the material in http://svn.openfoundry.org/pugs/docs/other/porting_howto which I won't repost here. Regexs are one of the areas I'm most confident in (when the parser doesn't break ;) ).
An option is included to only change captures inside regexs and just add the :perl5 modifier instead of other translations.
Heredocs are translated.
Calls to the scalar function are now handled by +, so scalar@foo becomes +@foo.
?foo? becomes m:once/foo/
Anything exported or export_okay is now exported with the ... is export or ... is export(:MANDATORY).
Printing to a file now uses a colon after the file handle.
Variables no declared before use gets declerations, or, optionally, "no strict;" can be used.

This is, unfortunately, not everything needed to translate Perl 5 to Perl 6. However, it is a start (maybe even a good start).

I'm hoping that my progress is considerd good enough for Summer of Code work. I know that I haven't actually complteted my project, mostly because many things I thought were easy ended up hard, and only a few things I thought were hard ended up easy. While the translator isn't complete, I've still provided some good work to the Perl community, and I hope to continue my work, even if my project is not considered a success.

At the very least, I can say I'm proud of my work. I revised my work daily most of the time, and I've learned much more then I thought I would.

I've made every effort to make my code open to others. If anybody wants to join me hacking my translator, it should be well enough documented to make working on it a little easier.

From here, I'm planning to continue work on my translator. First priority is bug fixes and added features, but I'd also like to start work on an expanded version of http://svn.openfoundry.org/pugs/docs/other/porting_howto , since working with this file has shown me how incomplete it is.

Now that my Summer of Code project is (technically) over, I want to thank the Perl community for all the help and support. #perl6 has been particularly useful, as well as everybody I've talked to in the community. Unfortunately, the reality of college means I won't be able to keep up my current pace of development, but I'm not going to leave this code alone either. Before this project I had never been a part of an Open Source project to this degree, and the Perl community made diving into everything, from svn to ghc to anything, easy and interesting.

I'm going to make an effort to make some revisions in the remaining time before the "due" time for my Summer of Code work, but this is essentially my final post. I'll update (likely tomorrow) with any final revisions and thoughts.

Two Days to Go

2006-08-19T15:56:00.000-07:00

With only a couple of days left, here's a quick update before I get back to coding:

Testing has been broadened, covering what should be a representative sample of Perl 5 code, with good results. As expected, the percentage covered dropped a little across more code, but well over 95% of code is still parsed without errors (the actual coverage is ~98%+). I'm trying my hardest to get the last bits covered.

Few new translations were added this week. I've been feeding some code to Pugs, and the biggest problem tends to be with pre-decleration of variables. I've added an option to add "no strict;"
which should fix code, but "no strict;" isn't supported by pugs yet. I've been working on the mechanics of adding declerations, but haven't made enough progress for it to be viable yet. So, long story short, there's no real fix yet for the decleration of variables in files that don't already use strict.

I think that, in the long run, "no strict" is a good option since the presumption of the parser is that the Perl 5 code is valid and safe. For now, I'm working on a translation option to add "my" or "our" as needed to declare variables and make it strict.

With appropriate declerations, most translated code (including my classic test file, TestInit.pm) works fine in Pugs.

With my last two days, I'm going to try bug fixes, more testing, and hopefully get a variable decleration translation in.

STM Progress Report (9)

2006-08-15T18:13:00.000-07:00

Done since last report:

Merge trunk into branch again.
Code cleanup for eventual merge to trunk

Allow NCI methods to be specified as writable and thus disabled on read-only objects.

Fixed some memory leaks.

Future:

A bit more cleanup/fixes before merging (incl. settling vtable method read-onlyness)
More testing, bug finding.

The end is near...

2006-08-11T15:47:00.000-07:00

All prophecies of doom aside, this week has been fairly productive. I've got a few more translations taken care of , as well as bug fixes. Documentation has been updated to reflect some new functionality, including the now completed option to do minimal regex changes using the :Perl5 modifier.

The code now runs without errors over ~98% of perl5's t directory (the actual percent covered is something like 97.77777...). However, my main problem is the appearance of unknown nodes. Since all possible nodes are now explicitly named, I know that Unknown nodes are appearing due to certain functions being called in places they shouldn't (many of my functions use an unknown node as a non-terminal way of returning an error). My first task of next week is to get rid of this problem.

I'm getting ready to really work my butt off this final week. Starting Monday I'll be back at new Mexico Tech, which means better internet access and plenty of free time. Hopefully I can get some real quality work done in the next week, to shore up some points that are still weak.

I'll still be updating next week, and I plan to post the week after that with a final report of what I've accomplished. Still, at this point I'd like to thank the Perl community for the opportunity to get to do this. I've learned an amazing amount this summer, and I look forward to continuing to work on Perl (and Pugs, and Parrot, etc...). Immediately following the Summer of Code I plan to continue revising and improving my translator, and helping in other areas when I can.

Thanks again for this great opportunity. Now to code...

STM Progress Report (8)

2006-08-07T18:27:00.000-07:00

Done since last report:

Bug fixes: fix race conditions; memory leaks; really make dynoplibs work with threads (sort-of)
Make things work on Solaris/Sparc 64
Look at performance issues including instrumenting STM to optionally record statisitical information (not very productive; it appears the best way to improve the performance of STM is to improve the performance of GC/allocation)

Futureish:

More test cases for STM so more bugs can be found/removed
Fix shared objects + detached threads bug
Documentation lookover
Exceptions + STM runtime
Thread performance/bug/incompleteness issues?

Another Week of Translations

2006-08-03T14:49:00.000-07:00

This week I've broadened my testing quite a bit. I've run my translator across all of perl5/t, and regularly rerun it over t/op (since this directory has a fairly representative sample of perl 5 code). As of today, roughly 98% of perl 5 code translates with no major problems (the translations may be wrong and are likely incomplete, but the translator doesn't crash either). For now I'm working on the robustness of the translator more then translations, but I have added a few this week (the svn logs should have all the details).

With the end of the project in site, I'm beginning to worry about completion. However, it seems stupid to worry about how far I get: if I don't finish by the SoC end date, I just continue to work and don't worry about the money. If I do finish, then great. Either way, worrying about it now only slows me down, so I'm just plugging away.

My goals for the next week are some continued testing, to identify areas that are not as finished as they should be, and working on some of the conversions left to do. I also plan to reread the synopses, as just today I found a translation (an easy one, luckily) that I had missed in design but should be in the code.

STM Progress Report (7)

2006-07-30T18:45:00.000-07:00

Since last report:

Fix up runtime library more (including N-argument choice).
Workaround the inability to share constant tables.
Add support for putting objects (i.e. ParrotObjects) into STM-managed variables.
Fix MMD handling for [STM]Ref-like types.
Fix GC-related race.
Fix bug where initial value of STM-managed variable would not be shared.
Started documenting deficiencies and 'shortcuts' in the Parrot threading implementation.
Merged in Wednesday's trunk.

Futureish:

STM/general multithreading benchmarks/performance improvements.
Solutions to various object sharing issues? Speeding up shared object GC?
More tests?
More cleanup?

Weekly report -- Pugs bootstrap -- 7/24 ~ 7/29

2006-07-30T10:26:00.000-07:00

(copied from my blog)

Main goal of this week was to complete the Parsec emitter. However, due to some natural limitation, it's not possible to really "complete" it, I just try my best to complete as many components as I can and hope those be enough for writing complete Perl 6 grammar. The accepting grammar constructions implemented from last post are:

:sigspace option
complete \X syntax (but not \Xxxx)
numbered captures
subrule with parameters
non-capture group

And I mailed pmichaud++ about the different semantics of "non-backtrack" in Perl 6 rules and Parsec parsing strategy. The :ratchet option, which is turned on by rule and token, is to make backtrack over atom fail. But different branches are still tried even if first several atoms in one branch are matched. In Parsec, the whole parsing fails if it goes into some branch, consumes some tokens and is unable to go on further. However, this can be changed by wrapping the branch with "try," which, as the name tells, tries to match the branch but will try other ones if failed. Such action is like adding a "::" after each atom instead of adding ":", which is done by rule and token in Perl 6 grammar.

One way to solve it is to add "try" everywhere. But that means giving up the high-efficiency parsing provided by Parsec. When the grammar is not LL, it's unavoidable. Parsec performs best on LL grammar (from official page), so in this stage, I'll feed only LL grammar to it and no additional "try" is added.

Translation Update

2006-07-27T14:36:00.000-07:00

This week has been mostly about basic issues, with only a few translations completed. On the other hand, I have enough translations completed that it's not surprising, since most of the stuff I have left to do is the stuff that's the toughest for me.

Completed stuff this week:
-Heredocs now parse correctly and translate.

-Yaml blocks with the chomp modifier (|+) parse correctly, even inside heredocs.

-All node types now recognized.

The result of all this is that, as far as I know, my translator can now at least partially translate any P5 AST with no unknown nodes. I'm working on a test script to test this theory, but in my manual testing it seems to be true.

Now I just need to get the rest of these translations done...

STM Progress Report (6)

2006-07-23T13:47:00.000-07:00

Done since last report:

Support using loaded dynpmcs/dynoplibs in new interpreters
Improve performance by not freezing simple PMCs (e.g. Integers) and by avoiding creating metadata hashes when all the metadata can be inferred otherwise.
Fixed a waitlist-related deadlock (hopefully).
Add support for extracting the log of the current transaction enough to run stm_wait. This is sufficient to implement things like Haskell STM's orElse.
Write (in PIR) a runtime library to avoid code replication and provide a place to place exception handling code when the PDD is implemented

Disturbing discoveries:

Neither Parrot_clone(..., sub_pmc) nor VTABLE_clone(..., sub_pmc) do the Right Thing for copying subroutines into a new interpreter. [Specifically, the first doesn't copy segment information and the second copy-on-write shares strings and doesn't switch the namespace_stash pointer.] Worked around for subroutines stored in globals and arguments to the thread-starting function; not worked around for subroutines stored more deeply nested (at least not yet).

Future:

Cleanup for pending merge.
More tests of runtime library, including tests for TODO features (e.g. exceptions).
Good examples of things that make good benchmarks (possible inspiration includes this).
Performance issues?
Ponder solutions to the cloning issue and (related) ways to avoid unneeded copying.

Progress report -- Pugs bootstrap -- ~ 7/23

2006-07-23T09:03:00.000-07:00

(copied from my blog)

I finally escaped from the final exams and projects, and the unexpected busy early July. The first checkin of the Pugs::Emitter::Rule::Parsec module is on July 20th. It accepted and emitted correct Haskell code on the yada example given in the README of MiniPerl6 module. (By the way, the Pugs::Grammar::MiniPerl6 and Pugs::Compiler::Rule modules have been moved from pX/ to perl5/)

After three days' hacking, it now accepts a lot of rule constructions. Also, a test file was added. In Parser.Literal there are 10 parser routines, two of them take arguments (namedLiteral and possiblyTypeLiteral) which I currently don't know how to present in Perl 6 rule, one uses previous parsing state to decide next action (ruleWordboundary), all other seven can be easily generated. In fact, all seven of them are already in the test file, the result is tested by replacing existing code by the generated one proven that it gears Pugs' parser, too.

P.S. As the matter of fact, two of the ten parsers (ruleDot and ruleLongDot) use non-LL grammar so that a little modification by hand on generated code is needed.

Translation News

2006-07-20T14:02:00.000-07:00

This week was a little slow due to a flu I managed to catch. Despite a week of decongestants and coughing, I managed to make considerable progress on translating regular expressions. I wrote a (fairly simple) parsec parser that takes a Perl 5 regex and spits out as close to a Perl 6 regex as it can. Some of the translations will have to be in external functions, since the parser only sees the text string of the regex, not the quotes or any of the other externals.

The sheer number of completed items to show for this week makes it look like a VERY productive week, even if a lot of the translations are almost trivial to implement (translating one metacharacter is a lot like translating another metacharacter, for the most part). Still, I'll take my victories where I can, and marking off a large number of items from the Perl 5 -> Perl 6 translation guide this week felt great.

For the remainder of the week I'm working on finishing up regexs and some of the tougher Stage one conversions. I think, with some careful re-reading of Synopsis 5 I can take care of my remaining regex issues. Beyond that, I need to get a clear reading of what issues I've handled, what I've identified and still not completed, and what I just plain haven't touched on yet. I'm hoping to start next week with getting a clear view of what I've done and what I still have to due, and then dive headlong into what I still have to tackle.

STM Progress Report (5)

2006-07-14T07:44:00.000-07:00

Done since last report:

Merged trunk into the STM branch.
Supported atomic ops for Solaris/Sparc V8plus and probably Solaris/Sparc V9
Supported cloning existing classes into new threads
Supported cloning HLL namespace data into new threads
Fixed some thread+GC bugs
Added revokable locks for STM

In progress:

More interpreter cloning – e.g. loaded extension libraries

Near Futurish:

Start investigating performance issues.

Further Futurish:

Runtime support for STM (should wait for exception support to be stable)
Less hackish cloning of PMCs into new interpreters (issue: clones must be deep, cannot be copy-on-write unless multithreaded COW improves (can't produce a COW string header in the wrong interpreter), and should not actually clone shared PMCs)

Translation Update

2006-07-13T13:47:00.000-07:00

Some big changes in the code this week. I've ended up with more code just for parsing and other tasks then I expected, so I've packaged everything up into some nice little modules. I also decided my code had reached the point where it was a good idea to move from a function called in GHCi to a compilable program. I've also added command line switches for some of the debugging functions, such as a verbose mode and unknown node identification.

I've also updated all my documentation with a full break down of what the modules do and how to use them. The only thing lacking is a makefile (I've been using ghc -- make for now).

Also this week I've added a couple of new translations (including foreach loops) and some HereDoc support (which is broken at this point, but it doesn't perturb anything else).

The code is now more usable then ever, it can parse everything but heredocs, and it can translate a good bit too, plus it's more user-oriented (as opposed to the function version, which was more for debugging then anything).

Perl 6 DBI update

2006-07-08T12:38:00.000-07:00

There is quite a lot of progress since my last report, I'm happy to say.

I've turned my attention towards Error Handling & Management, which happened to be tricky for multiple reasons. JDBC is Java, so it uses Throwable objects to throw exceptions. Inline::Java, which JDBC.pm is based on, uses the caught() method, eval and $@ to pass these exceptions to the Perl 5 side. Then I needed to trap these exceptions in Perl6 land.

Thanks to audrey++ for implementing my requests, so that now Pugs can handle unclaimed dies from perl5 land, caught by try { }, pass $@ as $!, and deal with the contents of $! if it happens to be a reference to the Throwable object dealing with the given exception (test here).

Another reason Error Handling is tricky, is because JDBC doesn't know the concept of a 'current error', but DBI has ($h->err), so there is a dispatch layer needed to handle the Error handling methods. I've written the exception handling part to trap JDBC errors, and a basic set_err method, currently works with RaiseError on.

I've also updated the Statement class's execute method, so that it can deal with SELECT and non-SELECT statements now aswell. On non-SELECT statements it returns the updated row count, if available. On SELECT statements it returns a true value.

To be able to retrieve the results of a SELECT statement, I've implemented fetchrow_arrayref in the Statement class.

Next, I'll work on getting more done from error handling & management, and when it reached a more reasonable state I'll start working on implementing prepared statements, with placeholders.

POE tests status update

2006-07-07T02:30:00.000-07:00

Some things I've completed include:

tests for POE::Resource::FileHandles.
tests for POE::Wheel::ReadWrite
I found the GTK bug in the end: POE wasn't checking whether it was idle and setting up its GTK callback appropriately.

I haven't quite got all of these changes and a few smaller ones checked into the main repository yet, and at the moment I'm working on refactoring the POE::Wheel::Run tests with support for the new 'pty-pipe' feature.

Translation Progress

2006-07-06T14:20:00.000-07:00

Not much to report this week. With the Independence Day holiday here in the U.S. I wasn't able to do as much as I'd like. Got some new translations done, the list of conversions shows what's completed, and it's good to see a fair number of done items. I'm already looking ahead to the next round of translations, since completion of stage one seems to be getting close.

I found some errors with my parser this week while testing more files, mostly files from t/op/ of the Perl distro, version 5.9.3. I've compiled a few bug fixes that I need to do, I hope to have everything I've found so far fixed by next week so I can continue to broaden the number of files my translator can at least partially translate.

For next week: bug fixes in the parser, more stage one translations, put out list of stage two conversions for comment/review/advice.

STM Progress Report (4)

2006-07-05T23:15:00.000-07:00

Done since last report:

Thread death no longer frees shared object when thread is joined (not deteached).
Waitlists implemented for STM so retry-type functionality is possible.
Some threaded DOD bugs fixed.
Revokable locking in STM.
Globals cloning.

Near futureish:

Freeze/thaw on STM* types.
Atomic ops for sparc.
PIR macros/investigate exception-handling issues.
More interpreter cloning (and sharing?) stuff (e.g. HLL data, registered classes).
Checking/fixing divergence of STM branch from trunk.

Perl5 -> Perl6 Translation Progress

2006-06-29T14:02:00.000-07:00

Lots to report this week (fortunately). I've managed to get 1-2 translations live in svn every day this week. Here's the summary of (semi-)completed translations:

-When referencing a hash element: $hash{$something} -> %hash{$something}

-When referencing a hash element with a constant key: $hash{someword} -> %hash

-Similarly, accessing an array element: $array[$i] -> @array[$i]

-Conditional statements with the ? and : operators: a ? b : c -> a ?? b !! c

-Substitution regexs: $a =~ s/blah/blah blah/g -> $a ~~ s:P5:g/blah/blah blah/

I've been testing my code on more test documents, and so far so good. Some unknown node types, but for the most part the parser runs smoothly. TestInit.pm is almost completely translated, and I've been trying some things in /t/op. Cond.t was particularly useful for testing my translation of ? and :, it translates completely and correctly.

Current topics of work:

-Stage two design document (mostly regex changes). The regex changes should actually be very easy to implement. As soon as I finish the design I'm going to try implementing a few, for the most part they should be very fast.

-Heredocs. I hadn't realized that the AST treats heredocs a bit differently then most nodes (though it does make sense). I had though that heredocs would essentially be a normal openquote/closequote pair, but the different structure (the node may have both kids and literal information) makes it harder to handle. I started to code a fix today, but it may not be live for a bit.

-The rest of the stage one translations.

I'm fairly happy with my progress this week. I would've liked to have more translations finished, but at least I have plenty to show.

STM Progress Report (3)

2006-06-29T11:04:00.000-07:00

Done since last report:

Stop-the-world garbage collection (so shared objects should be collected).
Threading API cleanup:
- Create ParrotRunningThread PMC type to replace calling methods on the current ParrotInterpreter object.
- Remove defunct type-{1,2,3} functions.
- Create interpreter cloning function for which one can request to clone various amounts (API is there, but mostly unimplemented/untested).
- When cloning code segments, properly propagate global namespaces with subroutines.
Extra test cases for STM.
Simple deadlock detection for STM.

In progress:

Interpreter death issues.

Future-ish:

Atomic ops for sparc.
Waitlists for STM.
Finish interpreter cloning function -- also test cases for it.
PIR macros for STM so that there will be no problems when exception support is added.

Report from DBI

2006-06-28T16:38:00.000-07:00

The time since my last report went without too much progress I'm afraid to say, due to the end of my exam session I had to wrap up.

Nonetheless, I've written a logging prototype, which unfortunately can't be run on Pugs at the moment, but capabilities will arrive soon for that (.meta class, Role runtime "mixins").

In the meantime there is a temporary logging structure built in the code I've written so far, just to allow to get on with doing other things apart from logging.

Apart from this I've been looking at the Java JDBC, to widen my understanding about it and I've been thinking of how to get from what we have (JDBC), to what we need (DBI v1 API).

Filtration

2006-06-26T01:37:00.000-07:00

I checked in a bunch of changes yesterday:

Stub tests for POE::Wheel::Curses and POE::Wheel::ReadLine (but this one is behaving a little strangely under Gtk).
A big group of changes for all the filter tests. I was initially only going to change POE::Filter::{RecordBlock,HTTPD,Stackable}, but I ended up factoring out some common code so I revised the other tests too. I also fixed a couple of minor bugs in POE::Filter::RecordBlock.
Tests for POE::Resources.

Today I'm working on finding a bug with perl -e 'use Gtk; use POE; $poe_kernel->run', and writing tests for POE::Resource::FileHandles.

Progress Marches On

2006-06-22T15:19:00.000-07:00

I feel I've got some good stuff done this week. I still have some nagging yaml issues, but beyond that I can reconstruct an AST into a complete program, and I have at least one translation ready to go into svn.

I finally got stuff committed to svn after far too long, and got some great revisions from Audrey. I had a couple of messy if/else/if/else... statements that Audrey turned into cases. I knew cases would work better there, but I hadn't been able to get one to work. This project has really taught me the weak points in my practical knowledge of Haskell. At my school they use Lisp to teach functional programming, so my knowledge of Haskell was more theoretical then practical, as show by my problems getting a case to work correctly. Luckily I'm picking things up quickly.

Today I made a major breakthrough on how I represent the AST. Since I (will) have a number of functions that only work on one type of node for translations, I decided to abstract out nodes to a larger degree. A P5AST is now made up of AbstractNodes and LiteralNodes, with each node having a type to further narrow it down, such as (AbstractNode Op_leave []) or (LiteralNode PNothing "1" "#junk"). This allows me to abstract out functions quite a bit, it cut down the print tree function to 5 or so lines of code.

Again, my mistakes are teaching me a lot. With the stuff Audrey corrected in my code I can make further change with a better understanding. I just hope I don't feel too stupid after these revisions.

All of my work to date is available at http://svn.openfoundry.org/pugs/misc/pX/Common/P5_to_P6_Translation/ . translations.hs shows Audrey's optimizations, translate-sage.hs shows my new approach to the AST (these two will be combined to create an effective version with a better tree). AST_description sums up some details on the AST that I think are useful, mostly the stuff that I've worked out on my own. Details on the rest of the directory are summed up in the README.

Overall I feel this has been a slow week, but with some really good stuff happening in the last few days. Good stuff, as we head into the midterm review. I'd say, while my project has moved slower then I'd like, it's only going to get faster from here, and I'm on track to get at least the minimum translations done, probably more.