How to do a proper benchmark (can't believe I missed this!)

[Ed]

Some time ago I tried to make a Nexuiz benchmark but it was inconsistent because of not everything being specified in the .cfg's that were executed. I've just been reading through the DP readme and spotted something I didn't see before:

Code: Select all

-benchmark <demoname> runs a timedemo and quits, results of any timedemo can be found in gamedir/benchmark.log (for example id1/benchmark.log)

So, to do a decent, consistent benchmark, run Nexuiz with the following command line options:

Code: Select all

-game benchmark -benchmark demos/demo1

The first bit specifies a mod directory. This is simply so that a directory is created called benchmark in which the log will be written. Since benchmark/ is new, data/default.cfg will be used ensuring consistency. The second bit specifies the demo being used.

Fire it up and the demo will be run and Nexuiz will exit upon completion. Then look in benchmark/benchmark.log for the result. A config.cfg file will have been created too. Do not modify it and delete it upon a Nezuiz update.

So this is what I got with the Linux i686 SDL version:
result 1910 frames 51.8550000 seconds 36.8334780 fps min/avg/max: 2.4875622/36.8334780/111.1111111

[esteel]

Since benchmark/ is new, data/default.cfg will be used ensuring consistency. The second bit specifies the demo being used.
A config.cfg file will have been created too. Do not modify it and delete it upon a Nezuiz update.

Sorry but this is not true. benchmark/config.cfg will take data/config.cfg as basis if benchmark/config.cfg does not exist. But what should work is to to create a empty benchmark/config.cfg file and then start Nexuiz as you told. It will then find a config.cfg in the benchmark directory and not fallback to data/config.cfg. However as its a special directory its save to delete/use an empty config.cfg without a user complaining .
Also i would advice to delete / recreate the empty config.cfg when a new version is out.

[Ed]

Sorry but this is not true. benchmark/config.cfg will take data/config.cfg as basis if benchmark/config.cfg does not exist.

No, it will not read data/config.cfg. I've tried. I know because it uses completely new settings and writes a config.cfg in the benchmark/ directory that is different from data/config.cfg.

[esteel]

It did read the data/config.cfg settings for me.. In both windows and linux. I used the 2.0.1 prerelease but i do not think it is any different here.

[Ed]

Hold on, yes, you are right. I've been looking at completely the wrong file. Yeah, everyone do as esteel says.

[esteel]

Result: result 1910 frames 32.5010000 seconds 58.7674225 fps min/avg/max: 4.7619048/58.7674225/250.0000000

Though because of shaders and caching stuff its better to just start nexuiz with -game benchmark and then start tdem demo1 twice.
So the whole procedure would be: create Nexuiz/benchmark, create empty Nexuiz/data/benchmark/config.cfg, start Nexuiz with -game benchmark and start twice tdem demo1
As you can see the result fromt he second run is: result 1909 frames 31.1210000 seconds 61.3412165 fps min/avg/max: 17.8571429/61.3412165/250.0000000
Thats a bit better.

But i guess your method is easier:
create Nexuiz/benchmark, create empty Nexuiz/data/benchmark/config.cfg, start Nexuiz with -game benchmark -benchmark demos/demo1
It might work to create a simple bench.zip that does contain the file Nexuiz/benchmark/config.cfg read-only and scripts to start the benchmark..


UNTESTED!!! http://esteel.planetnexuiz.de/nexuiz/misc/bench.zip

[KadaverJack]

An autoexec.cfg with

Code: Select all

exec normal.cfg
exec default.cfg
tdem demo1
tdem demo1

might be better... (also untested)

[esteel]

Well a user might have strange settings in his config.cfg that interfer with a benchmark. a empty config.cfg i saver. And i am not sure if tdem twice in the autoexec.cfg will do the job. It right start and restart the second timedemo right after the first without waiting for it to finish..

[KadaverJack]

And i am not sure if tdem twice in the autoexec.cfg will do the job. It right start and restart the second timedemo right after the first without waiting for it to finish..

Yeah, that's true
Unfortunately mutliple -benchmark arguments don't work either...
An alternative would be to bind "tdem demos/demo1" to one of the F keys (the F work in the menu unlike other keys).

[Ed]

An autoexec.cfg with

Code: Select all

exec normal.cfg
exec default.cfg
tdem demo1
tdem demo1

might be better... (also untested)

I'd do that with quake.rc I think. The double benchmarking isn't that important really. I realise there's the shader compilation thing but if that is being done every time it is consistent. When you play the game normally the shaders need to be compiled so do it now too.