CSharpWasmBenchmark - Results
CSharpWasmBenchmark is a project that benchmarks the performance of C# when compiled to Webassembly. The rationale behind this analysis is to find out whether it is viable to run performance critical applications written in C# in the browser and how the browser performance differs from "native" performance.
The performance of the following are measured: C# Runtime, C# Wasm AOT Blazor, C# Wasm Interpreted Blazor, C Wasm and JavaScript. This C# code has been compiled with .NET 7.
Improvements can probably be expected to C# Wasm in the future.
The JavaScript code is directly translated (as well as possible) from the C# source code, which may create certain differences between the implications of the code that is executed. You should always compare the source codes! The comparison between the performance of different C# execution targets are likely more reliable than that of comparison with JavaScript. JavaScript was included in this analysis since currently it is the standard frontend development programming language.
Note: The benchmarks may have errors in them and the source codes may not be equivalent. However, the C# code is always executed from the same source and should thus be comparable. Additionally, since C# Wasm AOT and Wasm itself are still somewhat experimental technologies, these results may not reflect the performance of the finalized product.
Unfortunately Blazor Wasm threw some memory error for larger parameter values. I do not currently have the time to investigate this and as such I just decreased the magnitude of the parameters. Note that the executed code is just the same for C# runtime and both C# Blazor variants.
Conclusions
Platform | Total Time Spent in Benchmarks (ms) | Relative Performance |
---|---|---|
Loading... |
Table 1: The total time spent in all the benchmarks. This data can be interpreted to give some sort of an average estimate of how well the different platforms perform. However, the performance depends on the parameter values. See the figures below for a more detailed overview.
The performance of C# AOT compiled to Webassembly is often significantly better than that of C# Wasm Interpreted. Table 1 shows the total time spent in benchmarking each platform. Running the benchmarks in C# Wasm AOT takes approximately 2.69x longer than in the C# runtime and running the benchmarks in C# Wasm Interpreted takes approximately 33.19x longer than in the C# runtime. These values are rough estimates and depend on parameter values. See more details in the figures below.
Generally the performance improvement of AOT in the featured benchmarks lies within the range 2x - 5x, but in some cases even a 20x performance benefit can be seen. Strangely, for example, sorting doubles seems to be slightly slower with AOT code.
C# Runtime can in some cases perform over 30x (even faster in certain cases) faster than C# Wasm AOT. However, the performance is almost on par in some cases with C# Wasm AOT. Probably when no abstractions or framework calls are utilized. See for example the C# Wasm AOT performance difference between ArraySortInt (uses Array.Sort) and ArraySortIntQuick (uses a custom quicksort implementation). The performance of ArraySortIntQuick in C# Wasm AOT is about 20x better than that of ArraySortInt, even if they, in practice, do the same work. The performance difference between ArraySortInt and ArraySortIntQuick when executing under the C# Runtime is negligible.
The equivalent code executed as JavaScript is generally faster than C# Wasm AOT. However, if the code avoids relying on framework methods and calls (for example ArraySortIntQuick, NewtonsMethodSecondDegree and NewtonsMethodThirdDegree) then the difference is almost negligible.
Webassembly promises, more or less, that it can enable "native" performance within a browser. C# often runs within a virtual machine and can maybe thus not achieve the full potential of Wasm. However, the difference in performance between C# Runtime and C# Wasm AOT varies from benchmark to benchmark and does generally not live up to the promise of Webassembly. Even JS is often faster than C# Wasm AOT. With the benchmarks and assumptions of this project the conclusion is that C# Wasm AOT still has a long way to become a general and performant client side web programming platform. Hopefully this changes in the future.
Credits
This analysis initially created by Acmion.
Contribute to CSharpWasmBenchmark in it's GitHub repository. Consider leaving a like if you found this analysis useful, interesting or informative.
Thanks to gabrielgt for his contributions in updating the project to .NET 7 and the C Wasm implementation.
Thanks to Uno.Wasm.Bootstrap, a project that provides an understandable way of AOT compiling C# to Wasm that just works and has minimal dependencies. Uno.Wasm.Bootstrap is no longer used in this benchmark, but for a long time it was used for C# Wasm AOT.
Benchmarking Strategy
This section describes how the benchmarks are created and executed. All benchmarks are contained within some
categories, for example, there is an ArrayBenchmarks
and a ListBenchmarks
category.
BenchmarkDotNet was not utilized in this project, because I wanted to ensure that the C# and JavaScript code
and benchmarks are doing the same things. Additionally, BenchmarkDotNet uses some complex compilation strategies,
which may or may not work with C# Wasm AOT.
C# Benchmarks
The source code of the C# benchmark executer can be found here. The way a C# benchmark should be defined is:
-
Create a C# class
SomeBenchmark
, which inherits fromBenchmark
. - Override some describing properties.
-
Override the
Parameters
property, which is of typeint[]
.SomeBenchmark
is initialized and executed with each of the values ofParameters
. This means that the performance of something can be evaulated with several parameter values. For example, ifParameters = new int[] { 1, 10, 100, 1000 }
andSomeBenchmark
sorts a list of numbers, then the analysis can be evaluated for a list of length 1, a list of length 10, a list of length 100 and a list of length 1000. -
Override the
public void Initialize(int parameter)
method. The value ofparameter
is one value from theParameters
array. This method is used to initialize stuff needed for the benchmark, but the performance is not evaluated. -
Override the
public object Execute()
method. This method should execute the code that is to be benchmarked. The method returns anobject
, which main purpose should be to ensure that the compiler does not remove the code within the method. The performance of this method is evaluated. -
Register
SomeBenchmark
under aBenchmarkCategory
inBenchmarking/Core/BenchmarkCategory.cs
. Note that you may need to define an extra category.
JavaScript Benchmarks
The source code of the JS benchmark executer can be found here. The way a JS benchmark should be defined is:
-
Create a JavaScript class
SomeBenchmark
. Note that the class name should be the same as the C# class name. - No need to override describing properties. The values from C# are dynamically injected wherever needed.
-
No need to define a
Parameters
array. The value from C# is dynamically injected wherever needed. -
Define the
function Initialize(parameter)
function. The behavior should be the same as the C# equivalent. This performance is not evaluated. -
Define the
function Execute()
function. The behavior should be the same as the C# equivalent. This performance is evaluated. - No need to register this class anywhere. It is dynamically injected wherever needed.
Benchmark Plots
Each benchmark is executed a certain number of times for each parameter value. The mean execution times of these runs represent data points, while the standard deviations represent error bars.
Benchmarks
The benchmarks.
ArrayBenchmarks
The benchmarks related to this category.
ArraySortInt
Generates an array of random ints.
Sorts the array.
The middle value of the sorted array.
The length of the array that is sorted.
ArraySortDouble
Generates an array of random doubles.
Sorts the array.
The middle value of the sorted array.
The length of the array that is sorted.
ArraySortIntQuick
Generates an array of random ints.
Sorts the array with a custom implemented Middle Point Pivot Quicksort (source).
The middle value of the sorted array.
The length of the array that is sorted.
ListBenchmarks
The benchmarks related to this category.
ListSortInt
Generates a list of random ints.
Sorts the list.
The middle value of the sorted list.
The length of the list.
ListSortDouble
Generates a list of random doubles.
Sorts the list.
The middle value of the sorted list.
The length of the list.
ListInsert
Generates a list of random ints.
Inserts 20% more values to the beginning of the list.
The middle value of the list.
The length of the list.
ListDelete
Generates a list of random ints.
Deletes 20% of the values from the beginning of the list.
The middle value of the list.
The length of the list.
MathBenchmarks
The benchmarks related to this category.
SummationInt
Generates a list of random ints.
Sums each number of the list.
The result of the summation.
The number of ints to sum.
SummationDouble
Generates a list of random doubles.
Sums each number of the list.
The result of the summation.
The number of doubles to sum.
MultiplicationInt
Generates a list of random ints.
Multiplies each number of the list.
The result of the multiplication.
The number of ints to multiply.
MultiplicationDouble
Generates a list of random doubles.
Multiplies each number of the list.
The result of the multiplication.
The number of doubles to multiply.
NewtonsMethodSecondDegree
Solves a root of a second degree function with a certain number of iterations.
The found root.
The number of iterations.
NewtonsMethodThirdDegree
Solves a root of a third degree function with a certain number of iterations.
The found root.
The number of iterations.
StringBenchmarks
The benchmarks related to this category.
StringConcatenation
Concatenates a single character to a string a certain number of times.
The middle character of the concatenated string.
The number of characters to concatenate.
StringConcatenationWithBuilder
Concatenates a single character to a string a certain number of times using a StringBuilder.
The middle character of the concatenated string.
The number of characters to concatenate.
DictionaryBenchmarks
The benchmarks related to this category.
DictionaryAccessInt
Generates a dictionary of random ints as keys with random ints as values.
Randomly accesses half of the keys and sums their respective values.
The sum of the values of the accessed keys.
The number of items in the dictionary.
DictionaryAccessString
Generates a dictionary of random strings as keys with random ints as values.
Randomly accesses half of the keys and sums their respective values.
The sum of the values of the accessed keys.
The number of items in the dictionary.