![]() ![]() You're right though that if you are taking individual benchmarks in a bubble and you are trying to compare them with one another that is a totally invalid/useless comparison. Now we can go back and look really carefully at the dates/scores/system configurations and figure out what changed and why. That means it's not really a bug that these scores can change over time when we're crowdsourcing the benchmark. That's because we have tens of thousands of tests to work with (probably a dozen or so per day on average for years across various boards). On Pi Benchmarks we can actually go back and figure out when they changed gcc and the benchmarks started dropping for example. Pi Benchmarks is kind of crowd sourcing that for the storage aspect to the entire community. Having a lot of tests certainly helps get closer to the truth though. ![]() They can be invalidated on the same day if they're running slightly different operating systems or firmware versions. ![]() They can be invalidated by days from how fast these apt packages can change. You're exactly right and it's actually worse than you're saying. It's important to know why numbers differ and it's also important to realise when numbers made in different years are still valid or not (and that's something ignored in this SBC world almost everywhere - the basic rules of passive benchmarking simply do not apply since it's a true hobbyist's world) employees tweaked ThreadX with better DRAM initialisation/timings and once you apt updateafter a reboot memory access improved. ![]() And this is not since the hardware magically got faster but since RPi Trading Ltd. For example RPi benchmark scores usually get better over the years. This (ability to compare/validate results made in different years) is also the reason why ramlat and tinymembench are included since they often can explain why the actual benchmark numbers differ. Sometimes it goes into the opposite direction and with more recent GCC the benchmark scores drop (cpuminer for example on Debian 11).Īs such that's the huge advantage of those 7-zip scores even if they represent performance metrics the average SBC user can not make lot use of: they represent 'hardware performance' way more precise in situations where software changed significantly and as such it makes sense to compare scores made in different years ( not true for the majority of other passive benchmarking tools where numbers fluctuate over time 'by design'). Most other benchmarks generate numbers that will change once you or your OS's maintainers update OS or compiler version. This limited focus is far away from generating a general performance recommendation since based on the use case people are using their SBC they need to pay attention to totally different areas (floating point, crypto acceleration, 2D/3D graphics acceleration, video acceleration, even I/O can be way more important than any of the other areas).īut while 7-zip's focus is pretty limited it has one huge advantage: it generates scores that are persistent accross OS and compiler releases (at least true for the 16.02 version that's shipped with Debian/Ubuntu since years). In that sense sbc-bench is more of an environment to execute the actual benchmark (see Geekbench and Phoronix 'piggyback' modes).Īs for the few individual benchmarks chosen two are more load generators / stressors (cpuminer, stockfish) and there is actually just one that really generates numbers to compare and that's 7-zip focusing on the stuff I'm personally interested in: integer+memory since me being a server guy. IMO stuff like this needs to be closely monitored so you're able to throw away results that are invalid (and since benchmarking server/storage systems is part of my day job I can confirm that throwing away results instead of relying on them is the 'norm' or what we mostly do since something went wrong we didn't know upfront so we need to adjust setup/methodology and repeat). Same goes for Raspberries with inappropriate powering and here it's even worse: the main OS (ThreadX on the VC) silently drops ARM clockspeeds to 600 MHz while the Linux kernel running on the ARM core still happily reports to run at 1200/1500/1800 MHz. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |