Summer of Code 2006

Since last report:

• Fix up runtime library more (including N-argument choice).
• Workaround the inability to share constant tables.
  
• Add support for putting objects (i.e. ParrotObjects) into STM-managed variables.
• Fix MMD handling for [STM]Ref-like types.
• Fix GC-related race.
• Fix bug where initial value of STM-managed variable would not be shared.
• Started documenting deficiencies and 'shortcuts' in the Parrot threading implementation.
• Merged in Wednesday's trunk.
  

Futureish:

• STM/general multithreading benchmarks/performance improvements.
• Solutions to various object sharing issues? Speeding up shared object GC?
• More tests?
• More cleanup?
  

(copied from my blog)

Main goal of this week was to complete the Parsec emitter. However, due to some natural limitation, it's not possible to really "complete" it, I just try my best to complete as many components as I can and hope those be enough for writing complete Perl 6 grammar. The accepting grammar constructions implemented from last post are:

• :sigspace option
• complete \X syntax (but not \Xxxx)
• numbered captures
• subrule with parameters
• non-capture group

And I mailed pmichaud++ about the different semantics of "non-backtrack" in Perl 6 rules and Parsec parsing strategy. The :ratchet option, which is turned on by rule and token, is to make backtrack over atom fail. But different branches are still tried even if first several atoms in one branch are matched. In Parsec, the whole parsing fails if it goes into some branch, consumes some tokens and is unable to go on further. However, this can be changed by wrapping the branch with "try," which, as the name tells, tries to match the branch but will try other ones if failed. Such action is like adding a "::" after each atom instead of adding ":", which is done by rule and token in Perl 6 grammar.

One way to solve it is to add "try" everywhere. But that means giving up the high-efficiency parsing provided by Parsec. When the grammar is not LL, it's unavoidable. Parsec performs best on LL grammar (from official page), so in this stage, I'll feed only LL grammar to it and no additional "try" is added.

This week has been mostly about basic issues, with only a few translations completed. On the other hand, I have enough translations completed that it's not surprising, since most of the stuff I have left to do is the stuff that's the toughest for me.

Completed stuff this week:
-Heredocs now parse correctly and translate.

-Yaml blocks with the chomp modifier (|+) parse correctly, even inside heredocs.

-All node types now recognized.

The result of all this is that, as far as I know, my translator can now at least partially translate any P5 AST with no unknown nodes. I'm working on a test script to test this theory, but in my manual testing it seems to be true.

Now I just need to get the rest of these translations done...

Done since last report:

• Support using loaded dynpmcs/dynoplibs in new interpreters
• Improve performance by not freezing simple PMCs (e.g. Integers) and by avoiding creating metadata hashes when all the metadata can be inferred otherwise.
• Fixed a waitlist-related deadlock (hopefully).
• Add support for extracting the log of the current transaction enough to run stm_wait. This is sufficient to implement things like Haskell STM's orElse.
• Write (in PIR) a runtime library to avoid code replication and provide a place to place exception handling code when the PDD is implemented

Disturbing discoveries:

• Neither Parrot_clone(..., sub_pmc) nor VTABLE_clone(..., sub_pmc) do the Right Thing for copying subroutines into a new interpreter. [Specifically, the first doesn't copy segment information and the second copy-on-write shares strings and doesn't switch the namespace_stash pointer.] Worked around for subroutines stored in globals and arguments to the thread-starting function; not worked around for subroutines stored more deeply nested (at least not yet).

Future:

• Cleanup for pending merge.
• More tests of runtime library, including tests for TODO features (e.g. exceptions).
• Good examples of things that make good benchmarks (possible inspiration includes this).
• Performance issues?
  
• Ponder solutions to the cloning issue and (related) ways to avoid unneeded copying.

(copied from my blog)

I finally escaped from the final exams and projects, and the unexpected busy early July. The first checkin of the Pugs::Emitter::Rule::Parsec module is on July 20th. It accepted and emitted correct Haskell code on the yada example given in the README of MiniPerl6 module. (By the way, the Pugs::Grammar::MiniPerl6 and Pugs::Compiler::Rule modules have been moved from pX/ to perl5/)

After three days' hacking, it now accepts a lot of rule constructions. Also, a test file was added. In Parser.Literal there are 10 parser routines, two of them take arguments (namedLiteral and possiblyTypeLiteral) which I currently don't know how to present in Perl 6 rule, one uses previous parsing state to decide next action (ruleWordboundary), all other seven can be easily generated. In fact, all seven of them are already in the test file, the result is tested by replacing existing code by the generated one proven that it gears Pugs' parser, too.

P.S. As the matter of fact, two of the ten parsers (ruleDot and ruleLongDot) use non-LL grammar so that a little modification by hand on generated code is needed.

This week was a little slow due to a flu I managed to catch. Despite a week of decongestants and coughing, I managed to make considerable progress on translating regular expressions. I wrote a (fairly simple) parsec parser that takes a Perl 5 regex and spits out as close to a Perl 6 regex as it can. Some of the translations will have to be in external functions, since the parser only sees the text string of the regex, not the quotes or any of the other externals.

The sheer number of completed items to show for this week makes it look like a VERY productive week, even if a lot of the translations are almost trivial to implement (translating one metacharacter is a lot like translating another metacharacter, for the most part). Still, I'll take my victories where I can, and marking off a large number of items from the Perl 5 -> Perl 6 translation guide this week felt great.

For the remainder of the week I'm working on finishing up regexs and some of the tougher Stage one conversions. I think, with some careful re-reading of Synopsis 5 I can take care of my remaining regex issues. Beyond that, I need to get a clear reading of what issues I've handled, what I've identified and still not completed, and what I just plain haven't touched on yet. I'm hoping to start next week with getting a clear view of what I've done and what I still have to due, and then dive headlong into what I still have to tackle.

Done since last report:

• Merged trunk into the STM branch.
• Supported atomic ops for Solaris/Sparc V8plus and probably Solaris/Sparc V9
• Supported cloning existing classes into new threads
• Supported cloning HLL namespace data into new threads
  
• Fixed some thread+GC bugs
• Added revokable locks for STM

In progress:

• More interpreter cloning – e.g. loaded extension libraries

Near Futurish:

• Start investigating performance issues.

Further Futurish:

• Runtime support for STM (should wait for exception support to be stable)
• Less hackish cloning of PMCs into new interpreters (issue: clones must be deep, cannot be copy-on-write unless multithreaded COW improves (can't produce a COW string header in the wrong interpreter), and should not actually clone shared PMCs)
  

Some big changes in the code this week. I've ended up with more code just for parsing and other tasks then I expected, so I've packaged everything up into some nice little modules. I also decided my code had reached the point where it was a good idea to move from a function called in GHCi to a compilable program. I've also added command line switches for some of the debugging functions, such as a verbose mode and unknown node identification.

I've also updated all my documentation with a full break down of what the modules do and how to use them. The only thing lacking is a makefile (I've been using ghc -- make for now).

Also this week I've added a couple of new translations (including foreach loops) and some HereDoc support (which is broken at this point, but it doesn't perturb anything else).

The code is now more usable then ever, it can parse everything but heredocs, and it can translate a good bit too, plus it's more user-oriented (as opposed to the function version, which was more for debugging then anything).

There is quite a lot of progress since my last report, I'm happy to say.

I've turned my attention towards Error Handling & Management, which happened to be tricky for multiple reasons. JDBC is Java, so it uses Throwable objects to throw exceptions. Inline::Java, which JDBC.pm is based on, uses the caught() method, eval and $@ to pass these exceptions to the Perl 5 side. Then I needed to trap these exceptions in Perl6 land.

Thanks to audrey++ for implementing my requests, so that now Pugs can handle unclaimed dies from perl5 land, caught by try { }, pass $@ as $!, and deal with the contents of $! if it happens to be a reference to the Throwable object dealing with the given exception (test here).

Another reason Error Handling is tricky, is because JDBC doesn't know the concept of a 'current error', but DBI has ($h->err), so there is a dispatch layer needed to handle the Error handling methods. I've written the exception handling part to trap JDBC errors, and a basic set_err method, currently works with RaiseError on.

I've also updated the Statement class's execute method, so that it can deal with SELECT and non-SELECT statements now aswell. On non-SELECT statements it returns the updated row count, if available. On SELECT statements it returns a true value.

To be able to retrieve the results of a SELECT statement, I've implemented fetchrow_arrayref in the Statement class.

Next, I'll work on getting more done from error handling & management, and when it reached a more reasonable state I'll start working on implementing prepared statements, with placeholders.

Some things I've completed include:

• tests for POE::Resource::FileHandles.
• tests for POE::Wheel::ReadWrite
  
• I found the GTK bug in the end: POE wasn't checking whether it was idle and setting up its GTK callback appropriately.

I haven't quite got all of these changes and a few smaller ones checked into the main repository yet, and at the moment I'm working on refactoring the POE::Wheel::Run tests with support for the new 'pty-pipe' feature.

Not much to report this week. With the Independence Day holiday here in the U.S. I wasn't able to do as much as I'd like. Got some new translations done, the list of conversions shows what's completed, and it's good to see a fair number of done items. I'm already looking ahead to the next round of translations, since completion of stage one seems to be getting close.

I found some errors with my parser this week while testing more files, mostly files from t/op/ of the Perl distro, version 5.9.3. I've compiled a few bug fixes that I need to do, I hope to have everything I've found so far fixed by next week so I can continue to broaden the number of files my translator can at least partially translate.

For next week: bug fixes in the parser, more stage one translations, put out list of stage two conversions for comment/review/advice.

Done since last report:

• Thread death no longer frees shared object when thread is joined (not deteached).
• Waitlists implemented for STM so retry-type functionality is possible.
• Some threaded DOD bugs fixed.
• Revokable locking in STM.
• Globals cloning.

Near futureish:

• Freeze/thaw on STM* types.
  
• Atomic ops for sparc.
• PIR macros/investigate exception-handling issues.
• More interpreter cloning (and sharing?) stuff (e.g. HLL data, registered classes).
• Checking/fixing divergence of STM branch from trunk.