0.9.2+ New WML preprocessor

[silene]

The new preprocessor uses a recursive grammar. This grammar was posted on the forum some times ago, so I will just outline some differences between the old and the new preprocessor. All the old scenarios should still work (except for text domains, see below): the new preprocessor is supposed to be strictly more powerful.

The old one could not parse this sentence {MACRO1 ({MACRO2 (A B)} C)} because the first right parentheses would have matched the first left parenthesis. See bug #10995 for a real scenario example (Wasvoid).

The old preprocessor was also unable to handle nested conditional directives:

Code: Select all

#ifdef A
  #ifdef B
    xxx#else
    yyy#endif
#endif

If A was not defined, it would have simply output "yyy" (whatever B and without any error message) instead of outputting nothing as it could be expected.

Macro definitions are the only part of the grammar that is not recursive: you can't nest a #define...#enddef inside another one, and you can't quote or comment #enddef.

The #textdomain tracking doesn't follow my initial proposal, it is a bit clever now. Consider this example:

Code: Select all

#textdomain tda
#define A B
  _"Something in tda" + {B}#enddef
...
#textdomain tdb
message={A _"Something in tdb"}

The translation of "Something in tda" won't be taken in the "tdb" domain (where it will be included), it will be correctly searched in the "tda" domain (where it is first defined). Moreover, the argument "Something in tdb" of the macro won't be translated in "tda", but in "tdb" (where it is defined).

Finally, text domains are no more WML-scoped, they last as long as no new #textdomain directive has been encountered, and they get reset at end of file and end of macro.

[torangan]

That means for a file containing macros one can simply put a #textdomain at the top and everything defined in that file will be in that domain?

[silene]

That means for a file containing macros one can simply put a #textdomain at the top and everything defined in that file will be in that domain?

Yes.

[yann]

wmlxgettext has now been updated to this new behaviour: it should behave correctly when a #textdomain is included in a macro definition.

[Invisible Philosopher]

Some questions/wishes:

Code: Select all

#file a.cfg
#textdomain foo
{b.cfg}

Code: Select all

#file b.cfg
key=_"bar"

Does _"bar" get translated in textdomain foo?

Code: Select all

#file a.cfg
#define INCLUDE_./_SOMEWHERE
{./somewhere.cfg}#enddef

Code: Select all

#file b.cfg
{INCLUDE_./_SOMEWHERE}

Does the file get searched for from a.cfg's parent?
Note that if it does, it is possible and practical for a campaign to be completely unaware of which directory it is placed in. Otherwise, if for example it uses a separate directory for maps, the scenarios must use an 'absolute' (for Wesnoth) path to include things like that (which is indeed the current practice, though I don't like it).
Implementing {../} inclusion syntax is not a good soution to this problem, because it requires each file to know how deep it is in the directory hierarchy (which is likely to change during WML refactoring).
EDIT: wait... [binary_path] and [textdomain] tags still require absolute paths... hmm...

Code: Select all

{MACRO ({ANOTHER_MACRO argument})}

Are parentheses still required around that?

Code: Select all

#define USING_TEXTDOMAIN TEXTDOMAIN_NAME TEXT
#textdomain {TEXTDOMAIN_NAME}
{TEXT}#enddef

Does that work at all? Does it work without affecting the textdomain later in the file it is used in? Will the newline after the #textdomain line get destroyed so that the macro doesn't introduce any extra newlines when used?


I also wish I could have spaces in the names of macros

Code: Select all

#define (RITUAL_IMAGE_Void Master)

[silene]

Code: Select all

#file a.cfg
#textdomain foo
{b.cfg}

Code: Select all

#file b.cfg
key=_"bar"

Does _"bar" get translated in textdomain foo?

No, text domains do not propagate into substituted macros or files. If they did, then wmlxgettext would get the wrong domains and the campaign would be untranslatable. Since no text domain is present before bar, the default text domain applies (which happens to be "wesnoth").

Code: Select all

#file a.cfg
#define INCLUDE_./_SOMEWHERE
{./somewhere.cfg}#enddef

Code: Select all

#file b.cfg
{INCLUDE_./_SOMEWHERE}

Does the file get searched for from a.cfg's parent?

This part of the preprocessor was not changed: the file would be looked up with respect to the parent of b.cfg.

Code: Select all

{MACRO ({ANOTHER_MACRO argument})}

Are parentheses still required around that?

No, neither are they around quoted strings containing spaces.

Code: Select all

#define USING_TEXTDOMAIN TEXTDOMAIN_NAME TEXT
#textdomain {TEXTDOMAIN_NAME}
{TEXT}#enddef

Does that work at all? Does it work without affecting the textdomain later in the file it is used in? Will the newline after the #textdomain line get destroyed so that the macro doesn't introduce any extra newlines when used?

Yes, yes, no.

Hmm... the last "no" means that it won't work in fact. Because the newline is not destroyed, the parser will complain that there is a quoted string lying around in

Code: Select all

message={USING_TEXTDOMAIN the_domain _"the string"}

The version below may work though, but I can't test right now.

Code: Select all

message=""+{USING_TEXTDOMAIN the-domain _"the string"}

I could make it so that #textdomain eats the newline.

I also wish I could have spaces in the names of macros

Code: Select all

#define (RITUAL_IMAGE_Void Master)

Hmm... spaces are allowed at the time of macro substitution, so you could indeed do

Code: Select all

{(RITUAL_IMAGE_Void Master)}

But parentheses are not allowed at definition time, so the point is kind of moot.

The three wishes make sense to me, so I may modify the preprocessor (and the parser for the newline issue) if there is no objection. Speaking about new behavior: should the following lead to two macro arguments or only one (there is no space between the two parts)? At the moment, it is interpreted as two arguments, requiring parentheses for it to be interpreted as one single argument.

Code: Select all

{A {B}C}

[Invisible Philosopher]

Speaking about new behavior: should the following lead to two macro arguments or only one (there is no space between the two parts)? At the moment, it is interpreted as two arguments, requiring parentheses for it to be interpreted as one single argument.

Code: Select all

{A {B}C}

One argument. That is IMO much more obvious behavior; spaces should be and easily can be used to separate macro arguments.

Oh, and does this work now? (having a macro argument that happens to have the same name as another macro)

Code: Select all

#define X Y
{Y}#enddef
#define Y Z
#enddef
{X Q}

[silene]

Oh, and does this work now? (having a macro argument that happens to have the same name as another macro)

Code: Select all

#define X Y
{Y}#enddef
#define Y Z
#enddef
{X Q}

What do you mean? Both with the old and the new preprocessor, the result would have been Q. Isn't it what you would expect?

[Invisible Philosopher]

What do you mean? Both with the old and the new preprocessor, the result would have been Q. Isn't it what you would expect?

I was of the impression that it didn't work with the old preprocessor. I don't remember why I thought that. Now I don't know if it was true.

[Invisible Philosopher]

Using CVS, I have found some problems (and check out this patch of mine, which helped me pinpoint some of them):

Wesvoid.cfg, my campaign file in the normal place:

Code: Select all

{./Wesvoid/public.cfg}

The inclusion fails. According to the output, it thought it was using

line 1 /Users/isaac//Library/Preferences/Wesnoth/data/campaignsWesvoid/public.cfg 1 /Users/isaac//Library/Preferences/Wesnoth/data/campaigns/Wesvoid.cfg 30 data/game.cfg

I have hilighted the error: ./ is broken currently because it doesn't have a '/' there where it should. It should only erase the '.', not the '/', when processing that path.

Also according to the output, in themes/default.cfg, which did

Code: Select all

#define FONT_TINY
10 #enddef

,

Code: Select all

font_size={FONT_TINY}

turns into

Code: Select all

font_size=#line 17 data/themes/default.cfg 1 data/game.cfg 166 data/themes/default.cfg 1 data/game.cfg
#textdomain wesnoth
10 #line 166 data/themes/default.cfg 1 data/game.cfg
#textdomain wesnoth

so, it appears you will have to implement textdomain eating the newline or else all key={MACRO} constructs will be broken.

Also I have a comment which I don't know whether it's a problem: In (Campaign_Name).cfg, from my campaign template, this broke (and gave some nonsensical error messages much later because of it):

Code: Select all

{@campaigns/(Campaign_Name)/external_units.cfg}

Putting parentheses around the whole thing fixed it:

Code: Select all

{(@campaigns/(Campaign_Name)/external_units.cfg)}

And the result of this is quite a mess:

Code: Select all

#define USING_TEXTDOMAIN TEXTDOMAIN_NAME TEXT
#textdomain {TEXTDOMAIN_NAME}
{TEXT}#enddef

key={USING_TEXTDOMAIN wesnoth-test (_"blah blah")}

to

Code: Select all

#line 1 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg
#textdomain wesnoth
#line 4 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg

key=#line 2 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg 5 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg
#textdomain wesnoth
#textdomain {TEXTDOMAIN_NAME}
#line 5 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg
#textdomain wesnoth
_#line 5 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg
#textdomain wesnoth
"blah blah"#line 3 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg
#textdomain {TEXTDOMAIN_NAME}
#line 5 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg
#textdomain wesnoth

#textdomain wesnoth
#textdomain wesnoth

What's worse, it's different that the output of this, which should be identical (it has no parentheses around _"blah blah"):

#define USING_TEXTDOMAIN TEXTDOMAIN_NAME TEXT
#textdomain {TEXTDOMAIN_NAME}
{TEXT}#enddef

key={USING_TEXTDOMAIN wesnoth-test _"blah blah"}

goes to

Code: Select all

#line 1 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg
#textdomain wesnoth
#line 4 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg

key=#line 2 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg 5 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg
#textdomain wesnoth
#textdomain {TEXTDOMAIN_NAME}
_#line 5 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg
#textdomain wesnoth
"blah blah"#line 3 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg
#textdomain {TEXTDOMAIN_NAME}
#line 5 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg
#textdomain wesnoth

#textdomain wesnoth
#textdomain wesnoth

, I think.

[silene]

Using CVS, I have found some problems (and check out this patch of mine, which helped me pinpoint some of them):

Wesvoid.cfg, my campaign file in the normal place:

Code: Select all

{./Wesvoid/public.cfg}

The inclusion fails. According to the output, it thought it was using

line 1 /Users/isaac//Library/Preferences/Wesnoth/data/campaignsWesvoid/public.cfg 1 /Users/isaac//Library/Preferences/Wesnoth/data/campaigns/Wesvoid.cfg 30 data/game.cfg

I have hilighted the error: ./ is broken currently because it doesn't have a '/' there where it should. It should only erase the '.', not the '/', when processing that path.

Hmm... strange. I can't reproduce it, but it probably is because I use a zipped Wesnoth. I will have to take a look at it.

Also according to the output, in themes/default.cfg, which did

Code: Select all

#define FONT_TINY
10 #enddef

,

Code: Select all

font_size={FONT_TINY}

turns into

Code: Select all

font_size=#line 17 data/themes/default.cfg 1 data/game.cfg 166 data/themes/default.cfg 1 data/game.cfg
#textdomain wesnoth
10 #line 166 data/themes/default.cfg 1 data/game.cfg
#textdomain wesnoth

so, it appears you will have to implement textdomain eating the newline or else all key={MACRO} constructs will be broken.

Your preprocessed output is wrong. The preprocessor does not generate additional # characters. So the fact that #textdomain does not eat newline characters is irrelevant here (and same for #line).

Also I have a comment which I don't know whether it's a problem: In (Campaign_Name).cfg, from my campaign template, this broke (and gave some nonsensical error messages much later because of it):

Code: Select all

{@campaigns/(Campaign_Name)/external_units.cfg}

Putting parentheses around the whole thing fixed it:

Code: Select all

{(@campaigns/(Campaign_Name)/external_units.cfg)}

Your file name contains parentheses? Then it's natural it broke, since the parentheses have a special meaning when preprocessing a bracketed block. And indeed, putting parentheses around the whole text will fix it.

[Invisible Philosopher]

Your preprocessed output is wrong. The preprocessor does not generate additional # characters. So the fact that #textdomain does not eat newline characters is irrelevant here (and same for #line).

Well actually I did not change the preprocessor. I just replace '\376' with '#' in the output because '\376' is not such a good character to look at.

[Invisible Philosopher]

Never mind about the ./ problem, I figured out what I was doing wrong.

[Invisible Philosopher]

putting macro argument usages in parentheses is still necessary (using CVS of about an hour ago (I hope I actually was using it, not that I have any reason to believe I wasn't)). For an example of how it goes "wrong":

Code: Select all

#define TESTY ARG
{VARIABLE {ARG}}#enddef

	[message]#in an event
	speaker=narrator
	message="{TESTY (a b)}"
	[/message]

displays

Code: Select all

[set_variable]
name=a
value=b
[/set_variable]

whereas some sort of error would be expected, or if not an error, at least

Code: Select all

[set_variable]
name=a b
value=
[/set_variable]

[silene]

Yes, it is the expected behavior of macro argument substitution: they are purely textual, they don't carry any semantic. Now that I see your example, I understand this behavior could be quite error-prone. I have to take a deeper look, but I already know it won't be an easy fix: it requires context-sensitive substitution to add parentheses where it will make a difference. Another solution would be to switch to a more unified (macro argument / macro) substitution model; this is the fix I prefer.

But there is a drawback. When I rewrote the preprocessor, I tried as much as possible to not break any existing WML code. And as a matter of fact, none of the mainline campaigns had to be fixed, and the few user campaigns I had went along fine (UtBS had to be fixed, but it contained a typo the old preprocessor silently misinterpreted, so I still consider it a success). However, changing the way macro argument substitution is handled will break all the campaigns that have parentheses in the macro definitions in order to enforce argument locality.

Consequently, this is a strong incentive for me not to fix the problem, although I itch to. I will only do it if quite a few campaign designers ask me to.