LZMA2-compressed savegames

Brainstorm ideas of possible additions to the game. Read this before posting!

Moderator: Forum Moderators

Forum rules
Before posting a new idea, you must read the following:
Post Reply
Spixi
Posts: 87
Joined: August 23rd, 2010, 7:22 pm

LZMA2-compressed savegames

Post by Spixi »

Hi, I have seen that ai0867 has merged the file-compressor bzip2 to trunk. Why didn't you use LZMA2 (.XZ)

LZMA2 has a higher compression rate than bzip2, however it needs a little more memory.

The XZ container format is the standard package format in Slackware GNU/Linux and LZMA2 is also used by 7zip - the most popular free file compressor for Windows users (Linux users use XZ utils instead).

I tried to compress the same file with gz, bzip2 and xz and unfortunately bzip2 has the worst compression rate.

mari[1.10]$ du -b save_index*
415828 save_index
23317 save_index.bz2
23002 save_index.gz
15024 save_index.xz

So I would suggest to use LZMA2 instead of Bzip2 (Note: libzma is supported by many platforms. It is public domain, therefore it is free to use by everyone.)

The liblzma Source is obtainable here: http://tukaani.org/xz/
Attachments
save_index.tar
This tarball compares a savegame.
Contents:
- plain uncompressed
- gzipped file
- bzip2'ed file
- xz'ed file
(470 KiB) Downloaded 172 times
Last edited by Spixi on October 16th, 2012, 7:04 pm, edited 1 time in total.
User avatar
MetalKing
Posts: 197
Joined: July 8th, 2011, 11:34 am

Re: LZMA2-compressed savegames

Post by MetalKing »

I Pro that Matter and refer to my Arguments written here: DataCpmpression with XZ Utils

Small Note: WinRAR is afaik able to extract Files from XZ as well afaik by treating the Archive as 7z.
"Sir! We are surrounded by our enemies!" - "Excellent ! We can attack in every direction!"
"Make everything as simple as possible, but not simpler." -- Albert Einstein
No Source - No Binary - No Trust!
Map Wesnoth Springs - The great War [200x120],Player=9
Max
Posts: 1449
Joined: April 13th, 2008, 12:41 am

Re: LZMA2-compressed savegames

Post by Max »

bzip2 came more or less for free as part of boost libraries. i'm pretty sure integrating lzma2 would be quite some effort.
Spixi
Posts: 87
Joined: August 23rd, 2010, 7:22 pm

Re: LZMA2-compressed savegames

Post by Spixi »

An implementation and integration of LZMA2 compression into GIMP only took 238 lines see here I don't think.that there is much efford ...
User avatar
ancestral
Developer
Posts: 1108
Joined: August 1st, 2006, 5:29 am
Location: Motion City

Re: LZMA2-compressed savegames

Post by ancestral »

You can certainly submit a Feature Request.

Meanwhile, patches are always welcome.
Wesnoth BestiaryPREVIEW IT HERE )
Unit tree and stat browser
CanvasPREVIEW IT HERE )
Exp. map viewer
Spixi
Posts: 87
Joined: August 23rd, 2010, 7:22 pm

Re: LZMA2-compressed savegames

Post by Spixi »

I filed a feature request in the ticket system of Boost here. Maybe - if this would be implemented - it could be used in a later release of BfW.
Max
Posts: 1449
Joined: April 13th, 2008, 12:41 am

Re: LZMA2-compressed savegames

Post by Max »

Spixi wrote:An implementation and integration of LZMA2 compression into GIMP only took 238 lines see here I don't think.that there is much efford ...
have a closer look. this commit relies on an external library and a couple of header files.
Spixi
Posts: 87
Joined: August 23rd, 2010, 7:22 pm

Re: LZMA2-compressed savegames

Post by Spixi »

Maybe we could use the Boost Lzma implementation from fusecompress ...
Max
Posts: 1449
Joined: April 13th, 2008, 12:41 am

Re: LZMA2-compressed savegames

Post by Max »

again - they are just using an external library.

that means this library would have to exist on all available platforms, work for all compilers. additional external dependencies put a heavy burden on maintainers and developers.
User avatar
MetalKing
Posts: 197
Joined: July 8th, 2011, 11:34 am

Re: LZMA2-compressed savegames

Post by MetalKing »

Spixi wrote:Maybe we could use the Boost Lzma implementation from fusecompress ...
Is their LZMA=LZMA2 ?
Max wrote:again - they are just using an external library.

that means this library would have to exist on all available platforms, work for all compilers. additional external dependencies put a heavy burden on maintainers and developers.
What Service does the external Lib provide? The Compressor should't do a Lot other than read and write a File and do internal Work with other Libs. I wonder ..

Is XZ implemented in C or C++? I prefer the OOP Way but Wrap is sweet like Sugar not Food.
"Sir! We are surrounded by our enemies!" - "Excellent ! We can attack in every direction!"
"Make everything as simple as possible, but not simpler." -- Albert Einstein
No Source - No Binary - No Trust!
Map Wesnoth Springs - The great War [200x120],Player=9
User avatar
Iris
Site Administrator
Posts: 6723
Joined: November 14th, 2006, 5:54 pm
Location: Chile
Contact:

Re: LZMA2-compressed savegames

Post by Iris »

Whether it is using C or C++ is not terribly relevant since C++ code can call C code just fine most of the time, and that is in fact how a lot of functionality is implemented (besides calls to system services). For example, Wesnoth uses a bundled version of the Lua interpreter, which is all done in C although it is built using a C++ compiler for compatibility reasons.
Author of the unofficial UtBS sequels Invasion from the Unknown and After the Storm (now available for Wesnoth 1.14.x and 1.15.4+).
Spixi
Posts: 87
Joined: August 23rd, 2010, 7:22 pm

Re: LZMA2-compressed savegames

Post by Spixi »

liblzma is a small file compression library. You can check for it in a configure script and only compile lzma-relating code if the library is found (e. g. with a define like HAVE_LZMA).

The main difference between LZMA and LZMA2 is, that the latter handles uncompressable files (for example some megabytes from /dev/urandom) better.

The original LZMA2 implementation used in XZ Utils is written in C. The LZMA SDK from 7-Zip also comes with C++ code
User avatar
MetalKing
Posts: 197
Joined: July 8th, 2011, 11:34 am

Re: LZMA2-compressed savegames

Post by MetalKing »

Are there any significant Constraits for Integration? I don't see any Reason against using the better Compressor and don't expect Problems with Integration. If the necessary Effort is lesser than writing a Posting I won't say anything about that Matter.
"Sir! We are surrounded by our enemies!" - "Excellent ! We can attack in every direction!"
"Make everything as simple as possible, but not simpler." -- Albert Einstein
No Source - No Binary - No Trust!
Map Wesnoth Springs - The great War [200x120],Player=9
Post Reply