Blackbox performance testing

1. The purpose of this test

I was reading the review for the new VectorLinux over at Mad Penguin and noticed how it also spoke of the new E17. As I was reading I came across a performance graph which piqued my curiosity. I followed the reference and got to Rasterman, who was kindly offering his testing program (wm_torture) under GPL.

Now, I don’t have a 3.4 GHz P4 handy to test on, I only have my 1.5 GHz Athlon XP, but I’ve decided to give it a shot and see how Blackbox does. Perhaps the most important goal of Blackbox is performance, so it’s only natural to be curious. If it behaves well, good for it. If it doesn’t, why not?

I’ve also tested some other managers while I was at it, to have some figures to relate to.

2. Disclaimer

Before you go look at the results, let me say some things. First of all, don’t bitch about the results. I’m posting the test conditions below. Some managers may have benefitted from a more recent version, but this is what I had around. I’m not going to go install the latest and greatest KDE for this test alone.

Second, this entire test was just meant to satisfy my curiosity. The results don’t mean that much in real life anyway. Raw power in mapping tens of windows very fast isn’t exactly what a desktop user looks for in their window manager.

Third, wm_torture only tests certain window operations. Read Rasterman’s post regarding this.

3. Test conditions

The test box has an NForce2 chipset, Athlon XP 1800+ (1533 MHz) CPU, 256 MB of RAM, a GeForce4 with Nvidia drivers installed, and X.org 6.8.2.

For each manager, I’ve started a clean X session, using only an aterm. I then started the manager and gave it (coughkwincough) enough time to load everything it needed to load. Then I ran the tests, each repeated 10 times in a row. For each manager I’ve removed any configuration I had out of the way, to allow it to run with default settings.

4. Results

Here are the results:

Response time graph
Throughput graph

(Props to Image Graph for the graphs.)

Response test results (lower is better)
===================================================

Results for: Blackbox - 0.70cvs
             MIN: 0.001735s, MAX: 0.004341, AVG: 0.002169
             MIN: 0.001605s, MAX: 0.003765, AVG: 0.002248
             MIN: 0.001649s, MAX: 0.003506, AVG: 0.002155
             MIN: 0.001624s, MAX: 0.003788, AVG: 0.002225
             MIN: 0.001915s, MAX: 0.003659, AVG: 0.002225
             MIN: 0.001905s, MAX: 0.003464, AVG: 0.002218
             MIN: 0.001908s, MAX: 0.003435, AVG: 0.002216
             MIN: 0.001910s, MAX: 0.003598, AVG: 0.002220
             MIN: 0.001906s, MAX: 0.003444, AVG: 0.002215
             MIN: 0.001907s, MAX: 0.003612, AVG: 0.002222

Results for: metacity - 2.8.8
             MIN: 0.003377s, MAX: 0.029933, AVG: 0.003508
             MIN: 0.002032s, MAX: 0.004464, AVG: 0.003426
             MIN: 0.003379s, MAX: 0.003718, AVG: 0.003439
             MIN: 0.003363s, MAX: 0.004509, AVG: 0.003423
             MIN: 0.003380s, MAX: 0.003672, AVG: 0.003440
             MIN: 0.003362s, MAX: 0.004473, AVG: 0.003420
             MIN: 0.003379s, MAX: 0.003758, AVG: 0.003436
             MIN: 0.003370s, MAX: 0.004501, AVG: 0.003430
             MIN: 0.003389s, MAX: 0.003721, AVG: 0.003446
             MIN: 0.003374s, MAX: 0.004516, AVG: 0.003432

Results for: Fluxbox - 0.9.13
             MIN: 0.004961s, MAX: 0.009186, AVG: 0.006807
             MIN: 0.006598s, MAX: 0.008345, AVG: 0.006687
             MIN: 0.006623s, MAX: 0.008394, AVG: 0.006718
             MIN: 0.006589s, MAX: 0.008288, AVG: 0.006681
             MIN: 0.006605s, MAX: 0.008364, AVG: 0.006709
             MIN: 0.006595s, MAX: 0.008384, AVG: 0.006693
             MIN: 0.006592s, MAX: 0.008322, AVG: 0.006682
             MIN: 0.006595s, MAX: 0.008354, AVG: 0.006690
             MIN: 0.006592s, MAX: 0.007019, AVG: 0.006684
             MIN: 0.006591s, MAX: 0.008344, AVG: 0.006691

Results for: Kwin - 3.0 (KDE 3.2.3)
             MIN: 0.009982s, MAX: 0.015350, AVG: 0.011487
             MIN: 0.009973s, MAX: 0.016063, AVG: 0.011585
             MIN: 0.011028s, MAX: 0.013697, AVG: 0.011560
             MIN: 0.009082s, MAX: 0.015970, AVG: 0.011569
             MIN: 0.009217s, MAX: 0.016399, AVG: 0.011578
             MIN: 0.009306s, MAX: 0.016375, AVG: 0.011579
             MIN: 0.011036s, MAX: 0.013110, AVG: 0.011571
             MIN: 0.011061s, MAX: 0.013703, AVG: 0.011568
             MIN: 0.011048s, MAX: 0.014568, AVG: 0.011572
             MIN: 0.008952s, MAX: 0.016450, AVG: 0.011577

Results for: Openbox - 3.2
             MIN: 0.002165s, MAX: 0.013044, AVG: 0.010194
             MIN: 0.008437s, MAX: 0.013791, AVG: 0.012115
             MIN: 0.008731s, MAX: 0.014198, AVG: 0.012395
             MIN: 0.008471s, MAX: 0.014426, AVG: 0.012569
             MIN: 0.008542s, MAX: 0.015255, AVG: 0.012760
             MIN: 0.009268s, MAX: 0.014623, AVG: 0.012852
             MIN: 0.009369s, MAX: 0.014857, AVG: 0.013014
             MIN: 0.009460s, MAX: 0.014814, AVG: 0.013078
             MIN: 0.009582s, MAX: 0.015676, AVG: 0.013201
             MIN: 0.009681s, MAX: 0.015152, AVG: 0.013284
Throughput test results (higher is better)
===================================================

Results for: Blackbox - 0.70cvs
             WIN/SEC 337.670700
             WIN/SEC 117.040507
             WIN/SEC 136.571722
             WIN/SEC 136.119966
             WIN/SEC 136.517002
             WIN/SEC 135.871226
             WIN/SEC 135.987524
             WIN/SEC 135.478961
             WIN/SEC 134.750369
             WIN/SEC 134.803227

Results for: metacity - 2.8.8
             WIN/SEC 54.943470
             WIN/SEC 54.535336
             WIN/SEC 53.976023
             WIN/SEC 51.653026
             WIN/SEC 51.236858
             WIN/SEC 51.085597
             WIN/SEC 50.687191
             WIN/SEC 50.598366
             WIN/SEC 50.811111
             WIN/SEC 50.708912

Results for: Fluxbox - 0.9.13
             WIN/SEC 47.255754
             WIN/SEC 62.134841
             WIN/SEC 46.452083
             WIN/SEC 41.660834
             WIN/SEC 39.435986
             WIN/SEC 44.448563
             WIN/SEC 43.533057
             WIN/SEC 56.662904
             WIN/SEC 37.148359
             WIN/SEC 41.312591

Results for: Openbox - 3.2
             WIN/SEC 41.259001
             WIN/SEC 26.357566
             WIN/SEC 33.470221

             WIN/SEC 27.278755
             WIN/SEC 30.385352
             WIN/SEC 30.267837
             WIN/SEC 32.421091
             WIN/SEC 27.213721
             WIN/SEC 26.324120
             WIN/SEC 31.017245

Results for: KWin - 3.0 (KDE 3.2.3)
             WIN/SEC 42.573518
             WIN/SEC 17.955763
             WIN/SEC 17.822661
             WIN/SEC 17.380420
             WIN/SEC 15.976924
             WIN/SEC 15.707538
             WIN/SEC 15.502637
             WIN/SEC 15.372216
             WIN/SEC 15.226521
             WIN/SEC 15.137349

I’ve sorted the managers from best to worst. The results speak for themselves. If you have issues with them, go do your own test.

5. My comments

I’m sorry Rasterman didn’t include Blackbox in his test, I trust it would have given E17 a good run for it’s money. The very good results weren’t that much of a surprise. If you build a manager around lightness and speed you’re probably going to get them.

As far as the other managers were concerned, I’ve tried to compare them to Rasterman’s test, assuming that figures should have doubled and halved, respectively, due to the twice as powerful CPU he used (P4 at 3.2 GHz).

But apparently all these managers are not very scalable. Fluxbox, KWin and Metacity doubled their WIN/SEC rate for him, but kept response times about the same. Openbox did it the other way around, halved response times for him but kept about the same WIN/SEC rate.

I’m not sure what to make of this. With twice the CPU juice you’d expect all around improvements. And if some managers can scale response time better and others WIN/SEC, you’d think it possible for one manager to scale everything better.