Improved units.cgi script

Jym · Post by **Jym** » March 19th, 2006, 11:14 pm

As we all know, the units.cgi script that automatically builds units advancement tree (http://wesnoth.slack.it/units.cgi) has some problems, among other it doesn't handle 1.1 WMLims for abilities so we cannot see units abilities...

And, what always annoyed me too, it only scans "base units" and not units specific to campaigns, this is not very problematic with official campaigns (since few units are specific to one of them) but might become problematic with UMC or, worse, with MP Eras (since to know what the advances of a unit are you need either to have see them in game or to go and look at the .cfg directly).

So, I get the script from penguin and made some work on it. Let's make things short and come directly to the good points:
_ The script is able to handle both 1.1 WML and 1.0 WML with satisfactory results.
_ The script can parse both the generic units and units specific to some official campaign, UMC, MP Era (you just need to tell which are to be parsed).
_ The script can now filter not only on the races but also on the campaigns or the MP factions, so you can tune display to exactely what you need.
_ The script can now be run completely standalone, so that you can have it on a local computer without internet connection (as long as you have set up a web server of course). This is especially good if, like me, you enjoy playing on your laptop in trains...
_ The script was tested on my linux and Mac OS X, I guess it should run reasonnably well on any unix system.

Among other things, the scanning of campaigns and filtering by campaigns means that we could have a database with every units in every UMC on the campaign server and links to the correct request from the campaign web page to see the new units in this campaign.
The 'standalone' stuff also mean that the complete database + images base can be build on a computer where everything (all UMC/Eras) is installed and up-to-date and then be uploaded to a web server where Wesnoth does not even have to be installed.

OK, so now let's come to the bad points (there are some):
_ MP Eras in 1.0 are not really well handled.
_ Don't look at the code, it's awful, not efficient and I am not really proud of that...

(these things happens when patching someone else code in a language you don't really know...)
_ One of the worst trick is the way to handle abilities by translating macros names (ABILITY_TELEPORT) into 1.0 like ability (ability=teleport). If someone describe a unit without using macros, it won't be parsed "correctly" (or even worse, if the macro name is not ABILITY_XXX)

Script can be downloaded from http://jymoyen.free.fr/Wesnoth/utils/units_tree.tgz
It contains a README that (hopefully) has enough informations in it for people who wants to use it.

Enjoy!

Post by **Soliton** » March 20th, 2006, 4:14 am

Thank you very much! I think this will a great help for many.

Let's see if we can get crimson penguin to upgrade units.wesnoth.org.

Oh and btw this also implements a feature request: https://gna.org/bugs/?func=detailitem&item_id=5453

Jym · Post by **Jym** » March 20th, 2006, 4:34 pm

Soliton wrote:Thank you very much! I think this will a great help for many.

Yes. Well, for me at least

Soliton wrote: Let's see if we can get crimson penguin to upgrade units.wesnoth.org.

I was discussing with him via private message and he cannot do that right now

Soliton wrote: Oh and btw this also implements a feature request: https://gna.org/bugs/?func=detailitem&item_id=5453

Cool! I'm even glader I've done it, then

Post by **turin** » March 21st, 2006, 2:07 am

Thanks a lot. Hope this gets added soon...

(I submitted the FR, BTW.

)

Post by **turin** » March 21st, 2006, 3:34 am

Hm. I have the Imperial Era installed, but when I cut out all lines in convert.conf similar to the Imperial_Era one except that line, it doesn't work.

The initial problem, I would suspect, is that you appear to reference the non-existent directory userdir/Imperial_Era. That doesn't exist. What DOES exist is userdir/data/campaigns/Imperial_Era/, which in turn has images, units, etc.

Post by **Viliam** » March 21st, 2006, 11:37 am

I know this would be too much work to do, but... how about also displaying localized names of the units? (Select languages by checkboxes, multiple languages allowed.)

IMHO this could be useful for translator, as many wesnoth units have very similar names -- so there is a risk of having translated two different units by same word. Also could be useful for non-English speaking players.

Jym · Post by **Jym** » March 21st, 2006, 5:56 pm

turin wrote:Hm. I have the Imperial Era installed, but when I cut out all lines in convert.conf similar to the Imperial_Era one except that line, it doesn't work.

The initial problem, I would suspect, is that you appear to reference the non-existent directory userdir/Imperial_Era. That doesn't exist. What DOES exist is userdir/data/campaigns/Imperial_Era/, which in turn has images, units, etc.

Normally, no.
I've used the stuff, exactly the convert.conf I've put on the web page, and I've had the imperial era (and the rest) found correctly.
The /data/campaigns/ part is added automatically by the script.
[reason : base units are in gamedir/data/units and not in /data/campaign/something/units so I prefered referring to the base dirs in both cases (gamedir and userdir) and let the script handle the stuff...]

Are you sure there was no stupid problem like having an extra space somewhere ? The line must be exactly "scan=userdir/Imperial_Era" with no extra spaces. Well, I guess that if you've not edited this line, this should be alright...

Do you have edited the early line in convert.conf which define the userdir ?
"userdir=/home/jym/.wesnoth" obviously if your login is not 'jym', this will not go to the correct dir

[and you also may want to edit the "gamedir=" line, mine is correct for the debian port of Wesnoth but not for other... Especially if you have Wesnoth for Mac, the path are completely differents... By the way, if you have Wesnoth for Mac, I do recommand to create symlink to the user/game dirs just to avoid spaces in the paths (I'm afraid those won't be dealed with correctly)]

If nothing work, you can try to edit convert.pl to add some debugging information. Between line 54 ("my $dir = shift;") and line 55 ("my $campname = "AAAAABase_units;"), add the following :
print "$dir\n";

Then, run convert.pl again. This should display where the script tries to find the campaigns and you'll be able to check if the information is correct at this point.

Hope this help. If not, bugger me again

(PM or email if you want)

Jym · Post by **Jym** » March 21st, 2006, 6:15 pm

Viliam wrote:I know this would be too much work to do, but... how about also displaying localized names of the units? (Select languages by checkboxes, multiple languages allowed.)

IMHO this could be useful for translator, as many wesnoth units have very similar names -- so there is a risk of having translated two different units by same word. Also could be useful for non-English speaking players.

You're right, this would be great...
The point is I don't know where the localisation stuff is stored...
OK, no, the real point is that the script now do what I needed and I'm probably too lazy to go on working on it...

If translation are stored like "Elvish archer=Archer elf" in a file, it should not be too hard to do...

Oh, by the way, if someone else want to work on the script, a nice to have would be the possibility to toggle display of units between "by races" (as currently done) and "by factions" in order to easilly see multi-racial factions (Knalgan, alliances of darkness/light, rebels, ...)

Post by **Darth Fool** » March 21st, 2006, 6:29 pm

Jym wrote: You're right, this would be great...
The point is I don't know where the localisation stuff is stored...
OK, no, the real point is that the script now do what I needed and I'm probably too lazy to go on working on it...

If translation are stored like "Elvish archer=Archer elf" in a file, it should not be too hard to do...

The localization stuff is stored in the po directory: a sample from po/wesnoth/it.po :

#: data/units/Orcish_Slurbow.cfg:3
msgid "Orcish Slurbow"
msgstr "Orco dalla Balestra Estesa"

of course, this is mixed in with all the other translation items.

Jym · Post by **Jym** » March 21st, 2006, 7:59 pm

Darth Fool wrote: The localization stuff is stored in the po directory: a sample from po/wesnoth/it.po :
#: data/units/Orcish_Slurbow.cfg:3
msgid "Orcish Slurbow"
msgstr "Orco dalla Balestra Estesa"
of course, this is mixed in with all the other translation items.

This looks like it should be doable...
However, I do not seem to have this po directory

(but I do have at least to languages). Apparently, debian handles things in a different way

The only po directory I have is for the South Guard (installed as an UMC since I don't have svn).

I have, however, some .mo files in /usr/share/locale/ They seem to contain the same information but in a much less convenient way (all the english first, then all the other language, in the same order).

Post by **torangan** » March 21st, 2006, 8:39 pm

The data is in the .mo files and those can be more easily accessed if you can find a modul to use gettext in your script. Then you'd only have to point it to the file and ask for the translation with the english name as a key...

Jym · Post by **Jym** » March 21st, 2006, 10:00 pm

OK, .mo seem indeed to be the good way to do the stuff... [gosh, I am again working on this script...]

And Gettext definitely exists for Perl, so this should be doable.

However, my poor knowledge in both Perl and Gettext (and internationalisation in general) just kicked in once more and I couldn't manage to make Perl Gettext work (in 1/2h).

Since I have something working with the .po files, I'll probably won't try harder with the .mo
However, if some Perl guru just gives me a complete example (say, the few lines needed to get the translation of "Mage" in french), I'll probably be able to infer how to do it and maybe even write it

Post by **turin** » March 21st, 2006, 10:55 pm

Here's a copy of my convert.conf file. I don't see any of the problems you mentioned... I'm using debian, BTW.

Interestingly, when I only change the listed install directories but leave it trying to make charts for the six different campaigns, it only does it for mainline.

EDIT

And, somehow, it works now.

Nevermind this, I guess. I have no idea what I did wrong.

Jym · Post by **Jym** » March 22nd, 2006, 12:15 am

Darn, one more evening spent coding for Wesnoth...
(I want to go back patching Slash'em !)

OK, so taking a few more things into account...

The script is now able to parse .po localisation files and display names of unit in several languages. I choose to display only the name for several reasons (other than usual lazyness).
_ because I think that the name of abilities/attack range/attack special are probably not the hardest thing to guess (there are few of them)
_ because Viliam only spoke about names

_ because this allow to store few extra info, thus allowing to have everything in the DB and so the script can be set on a computer where Wesnoth is not installed (the other solution being to have everything only in english and translate on the fly (Gettext ?) at display time rather than at convert time but htis require Wesnoth to be installed on the web server).

As mentionned, I only parsed .po files because I couldn't make Gettext for Perl work in the short tiem I tried. I someone gives me a complete example of Gettext for Perl, i'll probably try to work also with the .mo files...

This localisation stuff is not really tested since, as mentionned this evening, I only have the .po files for South Guard (yeah, I could have downloaded more for the sake of testing, right...)

Currently, if the unit name is different from its id, the localised names do not appear (since only name are translated, not id, so the .po contains names but the DB is sorted by id...)
I don't know how many units this affect... (2 in South Guard)
It is easy to correct this, but that will make the convert.pl script slower (probably something like taking twice as many time as now).
However, if the parsing of .mo works, this will be automatically solved...

By the way, I corrected a "bug" so that the tree now displays unit names and not ids.

And then, I added the possibility to display the list either by races or by factions, so you can see who is in the Knalgan alliance (or other). Selection by campaign applies for both and then you can display a certain number of races and/or a certain number of factions.

Turin, I lookd your convert.conf file. I can't figure what is wrong, but if the script finally worked, I think everything is more or less OK.

Oh, yes, the (updated) script is still at the same URL :
http://jymoyen.free.fr/Wesnoth/utils/

By the way, if someone has access to a web server that accept cgi scripts, maybe it would be a good idea to put the updated script with the DB build with 1 UMC, MP Era and 1 or 2 official campaigns, just so that people could look at what it now does without needing to set up a web server for themselves...
(I'm not completely sure that my provider allows cgi, or how he allows them... If I have a positive answer from their technical staff, I'll do it myself)

Post by **torangan** » March 22nd, 2006, 1:15 pm

Jym wrote:However, if the parsing of .mo works, this will be automatically solved...

That's not true. For getting the translation out of the .mo file you need the key string which is also in the .po file. The main problem with parsing .po files is that UMC do usually not deliver them. Those are only on WesCamp-i18n and the campaigns ship the .mo files required at runtime. Of course, there's also msgunfmt which makes a .po from a .mo.