Blackbox performance testing
- 1. The purpose of this test
- 2. Disclaimer
- 3. Test conditions
- 4. Results
- 5. My comments
1. The purpose of this test
I was reading the review for the new VectorLinux over at Mad Penguin and noticed how it also spoke of the new E17. As I was reading I came across a performance graph which piqued my curiosity. I followed the reference and got to Rasterman, who was kindly offering his testing program (wm_torture) under GPL.
Now, I don’t have a 3.4 GHz P4 handy to test on, I only have my 1.5 GHz Athlon XP, but I’ve decided to give it a shot and see how Blackbox does. Perhaps the most important goal of Blackbox is performance, so it’s only natural to be curious. If it behaves well, good for it. If it doesn’t, why not?
I’ve also tested some other managers while I was at it, to have some figures to relate to.
2. Disclaimer
Before you go look at the results, let me say some things. First of all, don’t bitch about the results. I’m posting the test conditions below. Some managers may have benefitted from a more recent version, but this is what I had around. I’m not going to go install the latest and greatest KDE for this test alone.
Second, this entire test was just meant to satisfy my curiosity. The results don’t mean that much in real life anyway. Raw power in mapping tens of windows very fast isn’t exactly what a desktop user looks for in their window manager.
Third, wm_torture only tests certain window operations. Read Rasterman’s post regarding this.
3. Test conditions
The test box has an NForce2 chipset, Athlon XP 1800+ (1533 MHz) CPU, 256 MB of RAM, a GeForce4 with Nvidia drivers installed, and X.org 6.8.2.
For each manager, I’ve started a clean X session, using only an aterm. I then started the manager and gave it (coughkwincough) enough time to load everything it needed to load. Then I ran the tests, each repeated 10 times in a row. For each manager I’ve removed any configuration I had out of the way, to allow it to run with default settings.
4. Results
Here are the results:


(Props to Image Graph for the graphs.)
Response test results (lower is better)
===================================================
Results for: Blackbox - 0.70cvs
MIN: 0.001735s, MAX: 0.004341, AVG: 0.002169
MIN: 0.001605s, MAX: 0.003765, AVG: 0.002248
MIN: 0.001649s, MAX: 0.003506, AVG: 0.002155
MIN: 0.001624s, MAX: 0.003788, AVG: 0.002225
MIN: 0.001915s, MAX: 0.003659, AVG: 0.002225
MIN: 0.001905s, MAX: 0.003464, AVG: 0.002218
MIN: 0.001908s, MAX: 0.003435, AVG: 0.002216
MIN: 0.001910s, MAX: 0.003598, AVG: 0.002220
MIN: 0.001906s, MAX: 0.003444, AVG: 0.002215
MIN: 0.001907s, MAX: 0.003612, AVG: 0.002222
Results for: metacity - 2.8.8
MIN: 0.003377s, MAX: 0.029933, AVG: 0.003508
MIN: 0.002032s, MAX: 0.004464, AVG: 0.003426
MIN: 0.003379s, MAX: 0.003718, AVG: 0.003439
MIN: 0.003363s, MAX: 0.004509, AVG: 0.003423
MIN: 0.003380s, MAX: 0.003672, AVG: 0.003440
MIN: 0.003362s, MAX: 0.004473, AVG: 0.003420
MIN: 0.003379s, MAX: 0.003758, AVG: 0.003436
MIN: 0.003370s, MAX: 0.004501, AVG: 0.003430
MIN: 0.003389s, MAX: 0.003721, AVG: 0.003446
MIN: 0.003374s, MAX: 0.004516, AVG: 0.003432
Results for: Fluxbox - 0.9.13
MIN: 0.004961s, MAX: 0.009186, AVG: 0.006807
MIN: 0.006598s, MAX: 0.008345, AVG: 0.006687
MIN: 0.006623s, MAX: 0.008394, AVG: 0.006718
MIN: 0.006589s, MAX: 0.008288, AVG: 0.006681
MIN: 0.006605s, MAX: 0.008364, AVG: 0.006709
MIN: 0.006595s, MAX: 0.008384, AVG: 0.006693
MIN: 0.006592s, MAX: 0.008322, AVG: 0.006682
MIN: 0.006595s, MAX: 0.008354, AVG: 0.006690
MIN: 0.006592s, MAX: 0.007019, AVG: 0.006684
MIN: 0.006591s, MAX: 0.008344, AVG: 0.006691
Results for: Kwin - 3.0 (KDE 3.2.3)
MIN: 0.009982s, MAX: 0.015350, AVG: 0.011487
MIN: 0.009973s, MAX: 0.016063, AVG: 0.011585
MIN: 0.011028s, MAX: 0.013697, AVG: 0.011560
MIN: 0.009082s, MAX: 0.015970, AVG: 0.011569
MIN: 0.009217s, MAX: 0.016399, AVG: 0.011578
MIN: 0.009306s, MAX: 0.016375, AVG: 0.011579
MIN: 0.011036s, MAX: 0.013110, AVG: 0.011571
MIN: 0.011061s, MAX: 0.013703, AVG: 0.011568
MIN: 0.011048s, MAX: 0.014568, AVG: 0.011572
MIN: 0.008952s, MAX: 0.016450, AVG: 0.011577
Results for: Openbox - 3.2
MIN: 0.002165s, MAX: 0.013044, AVG: 0.010194
MIN: 0.008437s, MAX: 0.013791, AVG: 0.012115
MIN: 0.008731s, MAX: 0.014198, AVG: 0.012395
MIN: 0.008471s, MAX: 0.014426, AVG: 0.012569
MIN: 0.008542s, MAX: 0.015255, AVG: 0.012760
MIN: 0.009268s, MAX: 0.014623, AVG: 0.012852
MIN: 0.009369s, MAX: 0.014857, AVG: 0.013014
MIN: 0.009460s, MAX: 0.014814, AVG: 0.013078
MIN: 0.009582s, MAX: 0.015676, AVG: 0.013201
MIN: 0.009681s, MAX: 0.015152, AVG: 0.013284
Throughput test results (higher is better)
===================================================
Results for: Blackbox - 0.70cvs
WIN/SEC 337.670700
WIN/SEC 117.040507
WIN/SEC 136.571722
WIN/SEC 136.119966
WIN/SEC 136.517002
WIN/SEC 135.871226
WIN/SEC 135.987524
WIN/SEC 135.478961
WIN/SEC 134.750369
WIN/SEC 134.803227
Results for: metacity - 2.8.8
WIN/SEC 54.943470
WIN/SEC 54.535336
WIN/SEC 53.976023
WIN/SEC 51.653026
WIN/SEC 51.236858
WIN/SEC 51.085597
WIN/SEC 50.687191
WIN/SEC 50.598366
WIN/SEC 50.811111
WIN/SEC 50.708912
Results for: Fluxbox - 0.9.13
WIN/SEC 47.255754
WIN/SEC 62.134841
WIN/SEC 46.452083
WIN/SEC 41.660834
WIN/SEC 39.435986
WIN/SEC 44.448563
WIN/SEC 43.533057
WIN/SEC 56.662904
WIN/SEC 37.148359
WIN/SEC 41.312591
Results for: Openbox - 3.2
WIN/SEC 41.259001
WIN/SEC 26.357566
WIN/SEC 33.470221
WIN/SEC 27.278755
WIN/SEC 30.385352
WIN/SEC 30.267837
WIN/SEC 32.421091
WIN/SEC 27.213721
WIN/SEC 26.324120
WIN/SEC 31.017245
Results for: KWin - 3.0 (KDE 3.2.3)
WIN/SEC 42.573518
WIN/SEC 17.955763
WIN/SEC 17.822661
WIN/SEC 17.380420
WIN/SEC 15.976924
WIN/SEC 15.707538
WIN/SEC 15.502637
WIN/SEC 15.372216
WIN/SEC 15.226521
WIN/SEC 15.137349
I’ve sorted the managers from best to worst. The results speak for themselves. If you have issues with them, go do your own test.
5. My comments
I’m sorry Rasterman didn’t include Blackbox in his test, I trust it would have given E17 a good run for it’s money. The very good results weren’t that much of a surprise. If you build a manager around lightness and speed you’re probably going to get them.
As far as the other managers were concerned, I’ve tried to compare them to Rasterman’s test, assuming that figures should have doubled and halved, respectively, due to the twice as powerful CPU he used (P4 at 3.2 GHz).
But apparently all these managers are not very scalable. Fluxbox, KWin and Metacity doubled their WIN/SEC rate for him, but kept response times about the same. Openbox did it the other way around, halved response times for him but kept about the same WIN/SEC rate.
I’m not sure what to make of this. With twice the CPU juice you’d expect all around improvements. And if some managers can scale response time better and others WIN/SEC, you’d think it possible for one manager to scale everything better.
