Yet Another Pugs Hacker's Blog

August 15th, 2006 -- Last Day of SoC

2006-08-15T13:52:00.000+08:00

In fact, this is a post for 8/14. I worked until three o'clock this morning.

Since fglock++ worked a lot on changing Pugs::Rule::Compiler to make module v6 better, I also made some changes to make my existing code work after the merge from pX/ to perl5/. After that, since most parts of Literal.hs are in the test file of the Parsec emitting module, I began writing the translation tool.

It is now here, with the Literal.grammar added. Since parameter of rules is not supported, three of them are still written in Haskell after the "use Haskell" statement. Beside that, a patch file is included which make the generated file really work.

The reasons of needing a patch file are: non-LL grammar which need backtrack and a weird error message without type annotation. There are three places where "try" is added in the generated source to allow backtrack, they can be avoid by rewriting the grammar.

But the need of annotation is not clear to me. The original parser is like this:

ruleTwigil :: RuleParser String
ruleTwigil = option "" . choice . map string $ words " ^ * ? . ! + ; "

And the generated version is (ugly and obviously not hand-written)

ruleTwigil = option "" $ do
string "^"
<|>
string "*"
<|>
string "?"
<|>
string "."
<|>
string "!"
<|>
string "+"
<|>
string ";"

Without type annotanion, both version of code cause the error

No instance for (MonadState RuleState (GenParser Char st))

Any comment?

July 31st, 2006 -- Parsec emitter and underlay problems

2006-07-31T00:44:00.000+08:00

Main goal of this week was to complete the Parsec emitter. However, due to some natural limitation, it's not possible to really "complete" it, I just try my best to complete as many components as I can and hope those be enough for writing complete Perl 6 grammar. The accepting grammar constructions implemented from last post are:

:sigspace option
complete \X syntax (but not \Xxxx)
numbered captures
subrule with parameters
non-capture group

And I mailed pmichaud++ about the different semantics of "non-backtrack" in Perl 6 rules and Parsec parsing strategy. The :ratchet option, which is turned on by rule and token, is to make backtrack over atom fail. But different branches are still tried even if first several atoms in one branch are matched. In Parsec, the whole parsing fails if it goes into some branch, consumes some tokens and is unable to go on further. However, this can be changed by wrapping the branch with "try," which, as the name tells, tries to match the branch but will try other ones if failed. Such action is like adding a "::" after each atom instead of adding ":", which is done by rule and token in Perl 6 grammar.

One way to solve it is to add "try" everywhere. But that means giving up the high-efficiency parsing provided by Parsec. When the grammar is not LL, it's unavoidable. Parsec performs best on LL grammar (from official page), so in this stage, I'll feed only LL grammar to it and no additional "try" is added.

July 23rd, 2006 -- Pugs::Emitter::Rule::Parsec

2006-07-23T23:58:00.000+08:00

I finally escaped from the final exams and projects, and the unexpected busy early July. The first checkin of the Pugs::Emitter::Rule::Parsec module is on July 20th. It accepted and emitted correct Haskell code on the yada example given in the README of MiniPerl6 module. (By the way, the Pugs::Grammar::MiniPerl6 and Pugs::Compiler::Rule modules have been moved from pX/ to perl5/)

After three days' hacking, it now accepts a lot of rule constructions. Also, a test file was added. In Parser.Literal there are 10 parser routines, two of them take arguments (namedLiteral and possiblyTypeLiteral) which I currently don't know how to present in Perl 6 rule, one uses previous parsing state to decide next action (ruleWordboundary), two use negative look-ahead (ruleDot and ruleLongDot) but <!before pattern> is not ready yet, all other five can be easily generated. In fact, all five of them are already in the test file, the result is tested by replacing existing code by the generated one proven that it gears Pugs' parser, too.

UPDATE: <!before pattern> support is added to both Pugs::Compiler::Rule and Parsec emitter. However, since the existing parser is not LL, I have to put an additional "try" in the generated code by hand to make it work.

June 5th, 2006 -- Named Captures

2006-06-05T22:31:00.000+08:00

Someone in another open source project I joined made some mistake and since the system is online, it corrupted many users config files which took me a lot of time help fixing it. (The error report is still growing..)

I added a rule to Pugs::Grammar::Rule so that it accepts more named capture syntax. It accepted only $<var> := (pattern). The parentheses was necessary. Now more general $<var> := [pattern] and $<var> := <subrule> are also valid.
Since when dealing with Perl 5 ratchet emitter, I've hacked the named capture emitting routine and made sure that it still work with the extended syntax. What made me surprised is, even the non-ratchet emitter works fine with the newly acceptable syntax. However, I'm still suspecting that the positional capture is not correctly generated. When $<var> := (pattern) is used, the (pattern) should NOT be listed in the positional captures.

Any way, the tests in ratchet shows that Pugs::Emitter::Rule::Perl5::Ratchet works correctly. And I'll add tests for the non-ratchet one if possible.

On the other hand, the Pugs::Emitter::Rule::Parsec plan has been in my brain for a while and there IS a Perl module for it now -- in my working copy and has a lot of functions with empty body. As soon as it has some basic construction, I will check it in and begin our Parser/*.hs replacement.

June 3rd, 2006 -- Well formatted grammar

2006-06-03T22:41:00.000+08:00

Still not begin writing Pugs::Emitter::Rule::Parsec. Still dealing with MiniPerl6. Aufrank++ and spinclad++ gave some great comments about my variable matching rule. And the post I sent to perl.perl6.language, which was NOT sent to the list since I posted it dirctly by group.google, was replied by pmichaud++, my mentor :)

The thread is here (http://0rz.net/7b1pZ). Following his comments, I changed the matching rule of <ws> and changed some rules into tokens. The grammar file is not so hateful now.

Update: Audrey++ confirmed that the PGE-way of <ws> is correct and has updated S05.

June 1st, 2006 -- A new month, a new stage

2006-06-01T17:02:00.000+08:00

Since the MiniPerl6 is finished, and it seems that there will not be big changes, I am planning moving on next stage, Pugs::Emitter::Rule::Parsec. There is no progress yet but I spent some time exploring papers on monads (most of them are published 10+ years ago) mostly for my current researching topic but also that Parsec is a monadic parser and I haven't totally understood monad.

On the other hand, I don't really like (pronounce "hate") the current MiniPerl6.grammar. It was well-formatted before :sigspace was implemented in Pugs::Compiler::Rule. But after the default-on option was implemented, I could not put extra spaces to format it well anymore. I concluded two reasons why it happened and posted a suggestion on perl.perl6.language but no one replied yet.

The two reasons (or, suggestion of changes) are:

Spaces at beginning and end of rule blocks should be ignored since space before and after current rule are most likely be defined in rules using current one. (The spaces around the rule-level alternative could also be ignored).
I am not sure the default rule of , I couldn't found it in S05. Currently the engine use :P5/\s+/ but I would like it to be :P5/\s*/ when it's before or after non-words and remains the same (\s+) otherwise.

I think these will help me formatting the grammar file better.

May 28th, 2006 -- MiniPerl6 Finished Again

2006-05-28T20:07:00.000+08:00

Continue on yesterday's work, I finished converting MiniPerl6 module so that it now work happily with ratchet.

The problem is that I used \w+ to match variable names but I should use [<alpha>|_]\w* instead (since the variables named with numbers are numbered captures, a special category). Another problem was hidden, or bypassed, by changing the order of alternations after I fixed the rule-using problem. It happened to be that the named capture parsing code in Pugs::Emitter::Rule::Perl5::Ratchet used a reference of boolean value in an "if" statement. I have to add another pair of ${} around it, at least before switching to Perl 6.

May 27th, 2006 -- Ratchet MiniPerl6

2006-05-27T21:39:00.000+08:00

The Pugs::Grammar::MiniPerl6 module was broken since the default setting of parsing option of Pugs::Compiler::Rule several days ago so that the ratchet option was turned off in it. Currently compile_p6grammar.pl generates a module used no ratchet feature which is identical to that it used to do. So that I focused on it these days.

I fixed the ratchet Perl 5 emitter on backslashes and named capture yesterday. And, I've almost finished fixing the MiniPerl6 module itself.

The main parts are to revise the grammar and update the translator compile_p6grammar.pl. The former one including changing some rules into "token" and rewrite the others to fit the :sigspace. And the later part is to make it accept "token" and "regex" as well as "rule."

Another big change is a modifier ":p" on rule ProductionRule. Since ProductionRule is the top rule in the grammar, the match of it should start from the very begin of input string. Correspondingly, compile_p6grammar.pl also translate it correctly.

I said that I "almost" finished fixing it. That means some of them are still broken and if you tried "make test" you will find the last test failed. I'll fix the temporary variable declaration as soon as possible. Stay tuned!

May 24th, 2006 -- Google Summer of Code!

2006-05-24T23:14:00.000+08:00

My proposal of "Pugs Self-hosting Bootstrap From Perl 5 and Rules" was accepted by google summer of code. There are five projects accepted from The Perl Foundation.

My proposal is listed

Shu-Chun Weng

Contact information
email: (hidden)
IRC: freenode.net #perl6 with id "scw"
Blog: http://YAPHB.blogspot.com

Perl 6 Self-hosting Bootstrap From Perl 5 and Rules

Synopsis
This is a subproject of Pugs (http://pugscode.org). One of the goals
of Pugs is to give an implementation of Perl 6 and help it
bootstraping. A new bootstrap plan has been proposed shortly. It
takes four steps and I propose to complete the first two steps in
this subproject. It consists of two new perl 5 modules. One
translates a very restrictive subset of Perl 6, MiniPerl6, to Haskell
and the another translates the Perl 6 Rule in to Parsec (a Haskell
parsing module) parsing code.

This is the easiest way to bootstrap we currently found. Bootstrap
from pure Perl 5 has been proven unbelievable hard due to the lack of
good parsing components. Pugs itself is written in Haskell because of
the Parsec module. It makes the parsing easier but still the parser
has grown too large to debug and maintain. Since the Perl 6 Rule has
been discussed for a long time and became stable and powerful enough,
the Perl 6 seems better be parsed by Perl 6 Rule itself.

Another feature of this project is that there is no "mixed" components
in the whole process. Bootstrap plans often consist of some parts
mixing two languages to get over the language gap before the target
can be used. But we carefully designed the plan and languages to be
used making that every components are written in a pure Perl 5 or Perl
6. This make the project reusable.

Deliverables
Module Pugs::Grammar::MiniPerl6
This module includes a Perl 6 Rule file and a Perl 5 script
translates the rule into Perl 5 parser module with the help of
Pugs::Compiler::Rule.

Module Pugs::Emitter::Rule::Parsec
This is the module used in the second step of bootstrap. It emits
Haskell code from the AST producted by Pugs::Compiler::Rule.

Module Pugs::Compiler::Rule
This is an existing module mainly written by Flavio S. Glock
(FGLOCK). Since it's heavily used is the above two modules and is
not completed, I may spend some time improving it to support more
Perl 6 Rule syntax.

Project Details
The new bootstrap plan is: (original in http://svn.openfoundry.org/pugs/misc/pX/Common/Pugs-Grammar-MiniPerl6/README)

1. This module, Pugs::Grammar::MiniPerl6, uses Pugs::Compiler::Rule
to read a special *mixed* Perl 6 Rule whose production rules
are written in Perl 5 (the current requirement of P::C::Rule).
1a. The rule is used to translate a subset of Perl 6, MiniPerl6,
to haskell.

2. Then, Pugs::Compiler::Emit::Parsec lands and uses this module
to translate the full Perl 6 grammar into a parser. The full
grammar can now write production rules in MiniPerl6 since
P::C::E::Parsec can use this module to translate such production
rules to haskell and makes the final output be pure Haskell.

3. When compiling Pugs, the .hs preprocessor will use
Pugs::Compiler::Emit::Parsec to accept the full Perl 6 grammar
generating Parser.hs. Then GHC will compile it to executable.

4. The executable can now read the full Perl 6 grammar again
generating compiler in PIL. Then self hosting is done.

I propose to finish the first two steps. Note that the first step
seems contain a mixed-language file, but since the Perl 5 code only
appear in production rules and are marked with 'use v5,' it's still
valid Perl 6.

The most challenging parts of this project are:

1. The parsed grammar saved in Perl 5 hash-and-array tree-like data
structure is extremely complex and nearly unreadable. Since the
correctness has not been tested much, debugging modules using it
is a difficult job.

2. We don't know if the Perl 5 Rule parser is powerful and efficient
enough to handle the full grammar. Currently translating a
sample grammar file with five small rules takes one minute CPU
time on a Xeon 2.8GHz machine (may be unprecise due to HTT).

3. We don't know if Parsec is powerful enough to handle all
functionality provided by Perl 6 Rules. Hopefully, we should be
able to use only those which can be translated easily in the
first version of the full grammar of Perl 6. And since Parsec
code was copied into Pugs' source tree for the ease of modifying,
it's not that serious.

The project has it's own directory in the Pugs' subversion repository
now:
http://svn.openfoundry.org/pugs/misc/pX/Common/Pugs-Grammar-MiniPerl6/

And the second part will appear as
misc/pX/Common/Pugs-Compiler-Rule/lib/Pugs/Emitter/Rule/Parsec.pm

The whole project will be licensed under the same licenses of the
Pugs source tree, which are currently dual licensed under GPL-2 and
Artistic-2.0b5.

Project Schedule
The following is the planned schedule:

May 31st -- Pugs::Grammar::MiniPerl6 has complete MiniPerl6 grammar
with correct production rules.
Jun 31st -- Pugs::Compiler::Rule be able to accept all syntax we
want to use in the full Perl 6 grammar.
Jul 1st -- Start organizing full Perl 6 grammar.
Jul 31st -- Pugs::Compiler::Emitter::Rule::Parsec can translate
most rule constructions.
Aug 1st -- Start rewriting exist parsing functions to Perl 6
rules.
Aug 20st -- Parser/*.hs can be replaced by the generated ones.
Sep 15th -- Parser.hs can be replaced by the generated one.

Bio
I am a undergraduated student interested in pragramming language
field as well as one of the commiter of Pugs. I use Perl a lot in
daily works (data processing, crontab tasks, etc.), and with the PL
interest, I noticed and took a look at Perl 6 about two years ago.
I joint Pugs project and became a commiter right after one month of
the birth of it. I focused mainly on parser and added some language
constructions.

Some works listed:

Analysis of ?? :: (currently ?? !!) posted on mailing list:
http://www.nntp.perl.org/group/perl.perl6.compiler/277

Some pugs blog entry mentioned my works
Array slicing http://pugs.blogs.com/pugs/2005/03/day_36_many_thr.html
debian control files and :key(val)
http://pugs.blogs.com/pugs/2005/03/day_56_leopold_.html
eval fix http://pugs.blogs.com/pugs/2005/03/day_54_yapctaip.html
http://pugs.blogs.com/pugs/2005/03/day_50_pugscc_r.html

Link to Further Information:
http://svn.openfoundry.org/pugs/misc/pX/Common/Pugs-Grammar-MiniPerl6/README

May 23rd, 2006 -- Ratchet rocks

2006-05-23T20:41:00.000+08:00

These days the Pugs repository was unusually silent, so did the channel which is full of joining and leaving messages. The main reason might be the disappearance of Audrey: her last commit was on May 18th, five days ago.

These days I spent most of development time on understanding and fixing ratchet. The goal was to make it handle compile_p6grammar.pl correctly. I achieved it several hours ago by properly handle match objects generated by quantified subrules. The result is:

$ time perl compile_p6grammar.pl MiniPerl6.grammar > MiniPerl6.pm
real    0m0.602s
user    0m0.516s
sys     0m0.027s

which took about two and half a minutes to complete, a two hundred times speed up. Amazing.

May 14th, 2006 -- Ratchet, don't backtrack

2006-05-14T22:32:00.000+08:00

As I mentioned several days ago, the Pugs::Grammar::MiniPerl6 was almost finished so that I turned my attention to Pugs::Compiler::Rule. fglock++ just committed the first version of ratchet parser which will not backtrack by default and will give an approximately 10x+ speedup by eliminating backtracking.

Currently the ratchet support is far from complete. 5 of 24 test cases failed before yesterday. I fixed one and found a serious bug on $pos resetting. I added two test cases for it so that it now fails 6 of 26 test cases.

fglock++ marked the failed tests as TODO and I found that I made a mistake on the meaning of "no backtracking," thanks to him and pmichaud++ correcting my understanding. I'll spend some time on the (?>) and section of backtracking of the perlre manual page.

May 10th, 2006 -- Variables! Tests!

2006-05-10T22:26:00.000+08:00

As I mentionedyesterday, Makefile.PL and a basic test file were added. But there were still something missing. I fixed the t/basic.t so that the test really passes. And the last two tests are corrected by updating the compile_p6grammar.pl to the one I used which deal with backslashes correctly.

On the another hand, I originally planned to support statements more than a single "return." But I dropped the idea due to two reasons:

I was not familiar enough to rules to write the grammar.

Translating general statements to Haskell is not straight forward.

But I implemented an alternative: temporary variables.

Assignment is not provided so variables can be translated to a simple "let ... in ..." expression, pure lambda one. The old yada example can now be written as

rule yadaLiteral {
   $<sym> := [ <'...'> | <'???'> | <'!!!'> ]
   {  my $yada = doYada( $<sym> );
      my $err = Val( VStr( $<sym> ~ " - not yet implemented") );
      return App(
       Var( $yada ),
       Nothing,
       [ $err ]
     )
   }
}

and translated into (the production rule part)

let yada = (doYada sym) in
let err = (Val (VStr (sym ++ " - not yet implemented"))) in
return $ (App (Var yada) Nothing [err])

Final notes:

Now it takes nearly three minutes for compile_p6grammar.pl to translate the grammar to .pm file.

There are three kinds of variables now: named capture, numbered capture, and temporary. Some renaming (prefix or suffix adding) may be performed to prevent conflict.

May 9th, 2006 -- Grammar and module formed

2006-05-09T23:10:00.000+08:00

It was really a long weekend and Monday. And a progressive Monday night and Tuesday. However, I forgot to write the blog entry last night. The numbered captures ($0, $1, ... ) are supported yesterday as well as escaped characters in string. But it didn't work until I commited the part of code dealing with character classes in Pugs::Emitter::Rule::Perl5 several minutes before. And the grammar now accept the array constructor "[1, 2, 3]".

clkao++ added the Makefile.PL and test suit making Pugs::Grammar::MiniPerl6 more like a module instead of the sank box of my own. I have to admit that I haven't written any perl module before and just studied the Makefile.PL in Pugs::Compiler::Rule.

The building process includes an invokation of compile_p6grammar.pl to translate the grammar file to the .pm file. I tried to use lrep instead but found character classes are not supported (just as what happened in Pugs::Compiler::Rule). But fglock++ stop me from switching to lrep and got offline right after that -- could you tell me why should I stay using compile_p6grammar.pl? :)

Currently all language constructions I planned to support in MiniPerl6 are implemented (except the "(a, b)" one but it's not necessary). I'll add more tests for it then move my concentration on Pugs::Compiler::Rule.

Updated: fglock++ left an IRC message to me "lrep is under a rewrite - compile_p6grammar.pl has less features, but it is more up-to-date". Ok, I'll use you script then :)

May 4th, 2006 -- A long silence, and function application

2006-05-05T00:36:00.000+08:00

The last post was on midnight of Sunday, 4 days ago. There were three in class midterms in 37 hours from Tuesday morning to noon of Wednesday. They are "special topics on graph algorithms," PDE, and compiler. Yes, I'm taking a compiler course this semester, which is a required course for junior. The exam problem is quite cool: given a grammar of Java-like (or may it's really a snip of java spec) array-newing syntax, write a parser accepting it, 100 points. No other constrains on algorithm. And here comes the bonus: write another one, 50 points. It took me some time to convince myself that write a parser in Perl 6 Rule will not count.

Then I wrote the proposal of summer of code and submitted it in evening. There are some more detail thought about the new plan with not many new ideas. But the may-faced challenging part is one of them:

The parsed grammar saved in Perl 5 hash-and-array tree-like data structure is extremely complex and nearly unreadable. Since the correctness has not been tested much, debugging modules using it is a difficult job.

We don't know if the Perl 5 Rule parser is powerful and efficient enough to handle the full grammar. Currently translating a sample grammar file with five small rules takes one minute CPU time on a Xeon 2.8GHz machine (may be unprecise due to HTT).

We don't know if Parsec is powerful enough to handle all functionality provided by Perl 6 Rules. Hopefully, we should be able to use only those which can be translated easily in the first version of the full grammar of Perl 6. And since Parsec code was copied into Pugs' source tree for the ease of modifying, it's not that serious.

clkao asked my plan on that if the second one really happened.

clkao> scw: what are you going to do if perl5 based rule parser is too slow?
scw> clkao: tune it :p
* clkao grins

I love the idea.

And some real-work reports. I fixed the variable and function naming rule so that it doesn't consume too many things in the afternoon. That makes function application as a operand of summation works. Then a little rewrite of rules of function application with parameters makes it work, too.

The performance problem seems affect my development more and more. I now do changing only on the generated Perl 5 module and only when everything works I'll merge the change to the grammar file and run the translation script again to see if the result is still correct. The reason is that I found the rule I used in the afternoon took one minutes and twelve seconds to translate and the one I committed several minutes before took nearly two minutes. It seems time to hack into Pugs::Compiler::Rule.

On the another hand, fglock++ revised the benchmark between P::C::R, inlined matching functions and Perl 5 regular expression. The performance seem to be ten times better one by one from the last one (P5regex) to the first one (P::C::R). Maybe I'll then write my own translation script in P5regex (a *lot* to do then) or inlined P::C::R structure instead of current one which uses P::C::R.

April 30th, 2006 -- Small improvement

2006-04-30T23:53:00.000+08:00

There is a course project deadline on midnight today (several minutes before). It took I and my partner about 50+ hours each. However, we've done it in time.

Since the rule ProductionRule consist of Statements and Return goes into infinite loop (I'm not sure, but I don't want to wait any more), I currently restrict it to have only a return statement. And I played with capture aliasing and quantified subpattern captures in Pugs::Compiler::Rule. It basically works but seems to have some problem on deeper matching?

I have a (self recursive) rule

rule AdditiveExpr {
    <FunctionAppExpr> <ws>?
    [ $<op> := (\+ | \- | ~) <ws>?
      $<rhs> := (<AdditiveExpr>) <ws>? ]?
    { use v5;
      if($<op>[0]){
          return "(" . $<FunctionAppExpr>() . " " . $<op>[0] . " " . $<rhs>[0] . ")";
      }else{
          return $<FunctionAppExpr>();
      }
    }
}

This works for matching "$<a> + t" there both terms can be matched by FunctionAppExpr, but when I feed "$<a> + t + 3", the "+ 3" part disappeared.

But some good news. r10165 can translate "return $<a> + t" to "return $ (a + t)", which valid Haskell and is what I want.

April 28th, 2006 -- The Grammar

2006-04-29T00:25:00.000+08:00

Came back from the future today, it's still April now, not May. Thanks TimToday.

I woke up really late today. It was about noon and I went to the office for the $part_time_job in the afternoon. But I can still spend some time on IRC. At about five o'clock, I fed the rule I drafted for MiniPerl6 to Pugs::Compiler::Rule and found some compilation error. Audrey pointed out that the comma may have to be escaped. That worked and many other problems are resolved in the same way including the plus sign, minus sign, tilde (string concatenation operator) and semicolon.

The draft of the grammar is now here. It's full of "<ws>?" since I do not yet know how to turn the word parsing mode on in P::C::Rule. And the string matching part lacks escaping sequence matching because Pugs::Emit::Rule::Perl5 currently ignores character class rule and I met some problem handling backslashes. I will spend some time to fix it. Also, production rules are mostly omitted but, as the sample parsing script demos, the written one works.

You might find that there is currently no Pugs/Grammar/MiniPerl6.pm in the repository. Yes, it's not there since it can be generated by feeding the grammar to the script under Pugs-Compiler-Rule. The automatic generating script will be added soon, maybe right after some kind of completion of the mini-grammar.

April 27th, 2006 -- The Plan

2006-04-27T22:56:00.000+08:00

I tried to feed the yadaLiteral rule describe yesterday (this morning) to Pugs::Compiler::Rule and found it failed to parse. The problem is that there is a $<sym> ~ ' - not yet implemented' in it but it's not valid perl 5 nor does the ad-hoc translator changed it correctly. But why is the production rule executed (eval-ed) before the rule is used to parse anything? Since the constructor did that.

I quickly wrote it down and planned to ask fglock, the main author of P::C::Rule, after dinner if s/he's online again. But when I attached the screen, the window eleven of the irssi is marked red and I found that audrey found this blog :) Welcome Audrey.

I had some thing to ask her, too. It's about the (just mentioned) perl 6 to perl 5 translator. After the conversation, the bootstrap plan forms (full text is here):

This module, Pugs::Grammar::MiniPerl6, uses Pugs::Compiler::Rule to read a special *mixed* perl 6 rule whose production rules are written in perl 5 (the current requirement of P::C::Rule).

The rule is used to translate a subset of perl 6, MiniPerl6,
to Haskell.

Then, Pugs::Compiler::Emit::Parsec lands and uses this module to translate the full perl 6 grammar into a parser. The full grammar can now write production rules in MiniPerl6 since P::C::E::Parsec can use this module to translate such production rules to Haskell and makes the final output be pure Haskell.

When compiling Pugs, the .hs preprocessor will use
Pugs::Compiler::Emit::Parsec to accept the full perl 6 grammar
generating Parser.hs. Then GHC will compile it to executable.

The executable can now read the full perl 6 grammar again generating compiler in PIL. Then self hosting is done.

That's all about today. There's a deadline this weekend so that there may not be any progress before Monday, sorry :(

April 26th, 2006 -- The Second Beginning

2006-04-27T12:15:00.000+08:00

I mainly focused on parser hacking in the development one year ago, so that this time when I asked Audrey, she remember that I've added the parsing function for :key(val) (an alternative pair constructing syntax) and "?? !!" ("?? ::" when I proposed it, in fact I didn't really added it. I wrote an analysis and she added it). Fortunately, she was refactoring Parser.hs preparing to replace it by a perl 6 rule file generated one.

So that's my task. I'm now writing a translator in perl 5 which accepts a perl 6 rule file (with some restrictions on the production rules). Here's an example. (source code of pugs can be obtained from http://svn.openfoundry.org/pugs/)

{- From src/Pugs/Parser/Literal.hs line 24 -}
yadaLiteral :: RuleParser Exp
yadaLiteral = expRule $ do
    sym  <- choice . map symbol $ words " ... ??? !!! "
    return $ App (Var $ doYada sym) Nothing [Val $ VStr (sym ++ " - not yet implemented")]
    where
      doYada "..." = "&fail_" -- XXX rename to fail() eventually
      doYada "???" = "&warn"
      doYada "!!!" = "&die"
      doYada _ = error "Bad yada symbol"

Can be written as (perl 6 rule format)

rule yadaLiteral {
   $<sym> := [ <'...'> | <'???'> | <'!!!'> ]
   { return App(
       Var( doYada( $<sym> ) ),
       Nothing,
       [ Val( VStr( $<sym> ~ ' - not yet implemented') ) ]
     )
   }
}

Now, there is already a perl 6 rule parser module written in perl 5: Pugs::Compiler::Rule. Currently it emits recursive-descent perl 5 parsing code. I'm going to add a Haskell backend for it.

< autreyt> so, your task, should you choose to accept it, is to use whatever the most expedient way to produce a parser for such a language, and emit Haskell code for it
< scw> in perl 5, but using three languages at once :D
< audreyt> :D
< scw> so what I could do is adding a backend for Pugs::Compiler::Rule
< scw> which targets on Haskell

How is the progress? Have you seen that it's April 27th now in +0800 time zone? What does that mean? Oh, nothing, just a joke. The idea of creating a new blog for this didn't come up until this morning. I finally understood how to use Pugs::Compiler::Rule last night, which was the goal I set for my self yesterday, then went back to the VFX ( Digital Visual Effects) project again. So that's all. I can now write a small peace of perl 5 code

use Pugs::Compiler::Rule;
my $rule = Pugs::Compiler::Rule->compile("moose { return 23 }");
print $rule->{perl5}

Which converts

rule foo{ moose { return 23 } }

into

do {
   package Pugs::Runtime::Rule;
   my $matcher =
      concat(
          constant( q!m! )
,         concat(
            constant( q!o! )
,           concat(
              constant( q!o! )
,             concat(
                constant( q!s! )
,               concat(
                  constant( q!e! )
,                   abort(
                      sub {
                          return { bool => 1, tail => $_[0], return => sub { return 23 } };
                      }
                  )
              )
            )
          )
        )
      )
 ;
 sub {
   my $grammar = shift;
   my $tree;
   rule_wrapper( $_[0],
       $matcher->( $_[0], $_[1], $tree, $tree, $grammar, 0, $_[0], $_[2] )
   );
 }
}

Scary, isn't it? However, my goal is

ruleFoo = do{ symbol "moose"; return $ VInt 23 }

Much clearer now! Hope I can, and see you next time.

Restart Hacking!

2006-04-27T11:54:00.000+08:00

Hello, I'm Shu-Chun Weng, a.k.a. scw. The commit bit was gained on March 1st, 2005 and I did some parser and builtin function hacks. But all these stopped on Apral 18th, 2005 due to the National University Sports Game hold on April 29th. There were two extra commits after then but those are not really "hacking."

However, after a whole year of stopping, I asked Autrey Tang if there's some thing I can help again, just as the first time I joint #perl6 and asked her (him, then). So that I'm now hacking *again*. *grin*