Optimising Nexuiz files

[Ed]

I have been experimenting with image optimisation and compression. I decided Nexuiz might be a good thing to have a go at as I can extract all of the files, optimise them with a set of shell scripts and then recompress. This is what I have done:

Code: Select all

jpegoptim --strip-all --quiet $a

Use JPEGoptim to remove unneccessary metadata and compact JPEG's losslessly.

Code: Select all

jpegtran -progressive

Convert all JPEG's to use progressive compression.

Code: Select all

jpegoptim --strip-all --quiet $a   # Optimise JPEG's losslessly
jpegoptim --strip-all -m95 --quiet $a   # Cap JPEG's at m%
jpegtran -progressive -outfile $a $a #.prog $a
jpegoptim --strip-all --quiet $a   # Optimise JPEG's losslessly

Some other JPEG stuff which limits quality to 95% (over that is generally pointless), then uses progressive compression, then does some more jpegoptim, for some complicated looping reason.

Code: Select all

advpng -z4

Use advpng to hack great lumps off of PNG's losslessly and quickly.

Code: Select all

optipng -zw32k -zc1-9 -zm1-9 -zs0-3 -f0-5 $b
optipng -zw16k -zc1-9 -zm1-9 -zs0-3 -f0-5 $b
optipng -zw8k -zc1-9 -zm1-9 -zs0-3 -f0-5 $b
optipng -zw4k -zc1-9 -zm1-9 -zs0-3 -f0-5 $b
optipng -zw2k -zc1-9 -zm1-9 -zs0-3 -f0-5 $b
optipng -zw1k -zc1-9 -zm1-9 -zs0-3 -f0-5 $b
optipng -zw512 -zc1-9 -zm1-9 -zs0-3 -f0-5 $b

Run 13608 different sets of compression parameters on all PNG's. This takes ages.

Code: Select all

zip -rq9DX

Zip each .pk3 or ZIP with the highest options.

Code: Select all

advzip ../$1 -z4

Recompact to attempt smaller filesize.

This leaves a completely lossless Nexuiz installation ZIP. This is unlike Nexuiz-lite which has files removed. On some JPEG's ~65% is saved.

The Nexuiz 1.5 download is normally 178.1Mb. I have got it down to 147.8Mb!

Again, this is entirely lossless, the game works exactly the same afterwards, no performance loss. The only downside is that it is 1bout an hour of computer time but it can all be automated.

I could send some of these scripts to the devs if you wish to reduce the Nexuiz download size. I think this might get more people downloading Nexuiz. All of the programs used are open source and I have downloaded them from various sources, latest versions.

I have also used similar algorithms on another game where I managed to take over 97% off of a JPEG, losslessly!

[Chozabu]

woah! that is kinda cool!

[tChr]

Of you do this on the JPEG that come with the game, you will hale loss.. You cannot alter a JPEG file (save it twice) without some amount of loss.. Hoever, if this is done to the oeiginal data set, itr great! Optimizations like this are always a good thing.

One question though: does the harder zip compression of the pk3 increase the loading time noticeable?

[tZork]

Convert all JPEG's to use progressive compression.

Dont, afaik progressive compression is useless for anything other then webbrowsers.

The advantage of progressive JPEG is that if an image is being viewed
on-the-fly as it is transmitted, one can see an approximation to the whole
image very quickly, with gradual improvement of quality as one waits longer;
this is much nicer than a slow top-to-bottom display of the image. The
disadvantage is that each scan takes about the same amount of computation to
display as a whole baseline JPEG file would.

http://www.faqs.org/faqs/jpeg-faq/part1/

[KadaverJack]

Dont, afaik progressive compression is useless for anything other then webbrowsers.

Yes, progressive encoding is pretty useless for Nexuiz, but it seems to have a interesting side effect: i've noticed that progressive jpegs are usually a bit smaller. But ony like 1 or 2KB in most cases, so it doesn't really matter.

[Ed]

Dont, afaik progressive compression is useless for anything other then webbrowsers.

It is not pointless. In every case I have seen it gives a smaller file size. Even if it is not, because of running jpegoptim (which doesn't support progressive JPEG's) afterwards, if it is smaller in a non-progressive form it would end up as non-progressive. It just always chooses the smallest option.

If a program does not support progressive JPEG's, it is all displayed in one slower pass. The increased number of passes is not really an issue nowadays. If you look at the webapages that say it is slower (including that one) they are all very old. That one, although dated 1999, seems a lot older in places. Maybe in 1992 the extra processing power mattered but it's insignificant now.

Handling the pk3 on loading will not take any longer. This is because 'slower' compression only takes longer on the compression run. Extracting any compressed file almost always take the same time, seeing how the file is now smaller, there is less disk access which would counteract any marginal increase in processing or memory bandwidth.

[tZork]

Im fully aware that this info is old but its still valid, and that the overhead alone may not be mutch/noticable, however everything adds up in the end and

The disadvantage is that each scan takes about the same amount of computation to display as a whole baseline JPEG file would.

can amount to a huge overhead with all the textures some maps need.

This vs 1-2k more on each file and ill go for the non-progressive anyday.

[Ed]

The progressive compression is the most contraversial bit of this as I expected. A lot of size loss can still be achieved using all of the other methods I've suggested. The progressive JPEG's probably only cut it down by 1Mb or so.

I would recommend that the PNG operations and recompression are implemented at least. The jpegoptim operations are not entirely lossless but would help. It depends what the dev team wants to do with these ideas. With 1.5.1 due sometime soon this could reduce a lot of bandwidth usage, afterall, there is no such thing as an 'unlimited Internet connection'. There are always limits.

[Ed]

Bump!

Any chance of applying some of these optimisations to 1.6? This could cut many Mb off of the downlaod. I've found that 7-zip can give better ZIP (not it's native 7z) file sizes than plain zip.

[leileilol]

progressive jpegs crash q3a engines too and choke the q3map compilers

(i know nexuiz isn't, but...)