CSharpWasmBenchmark - Results
CSharpWasmBenchmark is a project that benchmarks the performance of C# when compiled to Webassembly. The rationale behind this analysis is to find out whether it is viable to run performance critical applications written in C# in the browser and how the browser performance differs from "native" performance.
The performance of the following are measured: C# Runtime, C# Wasm AOT, C# Wasm Interpreted and JavaScript.
The C# Wasm compilation is handled by Uno.Wasm.Bootstrap, which is using .NET 6 preview technology. However, this may have some implications on the performance. .NET Blazor Interpreted gave runtime memory errors and Blazor AOT is not yet implemented. However, a limited comparison between Uno Interpreted and Blazor Interpreted is contained for some of the benchmakrs (prefixed with "Uno vs Blazor" in the plots). Generally, Uno is 2x faster than Blazor, but in some benchmarks the performance is the same.
The JavaScript code is directly translated (as well as possible) from the C# source code, which may create certain differences between the implications of the code that is executed. You should always compare the source codes! The comparison between the performance of different C# execution targets are likely more reliable than that of comparison with JavaScript. JavaScript was included in this analysis since currently it is the standard frontend development programming language.
Note: The benchmarks may have errors in them, the C# Wasm compilation may depend on stuff that is no longer considered best practice, the tools used in C# Wasm compilation may not be updated to their newest and best versions and the C# and JavaScript source codes may not be equivalent. Additionally, since C# Wasm is still a somewhat experimental technology these results may not reflect the performance of the finalized product.
Conclusions
The performance of C# AOT compiled to Webassembly is significantly better than that of C# Wasm Interpreted. Generally the improvement in the featured benchmarks lies within the range 2x - 5x better performance, but in some cases even a 50x performance benefit can be seen.
C# Runtime can in some cases perform over 100x faster than C# Wasm AOT and generally the performance benefit of C# Runtime is about 10x. However, the performance is almost on par with C# Wasm AOT when no abstractions or framework calls are utilized. See for example the C# Wasm AOT performance difference between ArraySortInt (uses Array.Sort) and ArraySortIntQuick (uses a custom quicksort implementation). The performance of ArraySortIntQuick is about 20x better than that of ArraySortInt, even if they, in practice, do the same work. The performance difference between ArraySortInt and ArraySortIntQuick when executing under the C# Runtime is negligible.
The equivalent code executed as JavaScript is generally 3x - 10x faster than C# Wasm AOT. However, if the code avoids relying on framework methods and calls (for example ArraySortIntQuick, NewtonsMethodSecondDegree and NewtonsMethodThirdDegree) then the difference is almost negligible.
The C# code was compiled with Uno.Wasm.Bootstrap, which uses .NET 6 preview tech. Uno Wasm interpreted is at best approximately 2x faster than Blazor interpreted in the test which were performed. In some cases the performance is the same. However, Blazor interpreted gave runtime memory errors and as such the validity of this conclusion is quite uncertain. Regardless, we can with some certainty say that the results of this analysis reflect the performance of Blazor to some extent.
Webassembly promises, more or less, that it can enable "native" performance within a browser. C# often runs within a virtual machine and can maybe thus not achieve the full potential of Wasm. However, the difference in performance between C# Runtime and C# Wasm AOT varies from benchmark to benchmark and does generally not live up to the promise of Webassembly. With the benchmarks and assumptions of this project the conclusion is that C# Wasm AOT (at least when compiled with Uno.Wasm.Bootstrap) still has a long way to become a general and performant client side web programming platform. Note: The usage of the Uno.Wasm.Bootstrap toolchain may have affected the performance of some of the benchmarks. Thus, this analysis should not be regarded as a benchmark of the finalized product. Hopefully this changes in the future.
Credits
This analysis was created by Acmion.
Contribute to CSharpWasmBenchmark in it's GitHub repository. Consider leaving a like if you found this analysis useful, interesting or informative.
Thanks to Uno.Wasm.Bootstrap, a project that provides an understandable way of AOT compiling C# to Wasm that just works and has minimal dependencies.
Benchmarking Strategy
This section describes how the benchmarks are created and executed. All benchmarks are contained within some
categories, for example, there is an ArrayBenchmarks
and a ListBenchmarks
category.
BenchmarkDotNet was not utilized in this project, because I wanted to ensure that the C# and JavaScript code
and benchmarks are doing the same things. Additionally, BenchmarkDotNet uses some complex compilation strategies,
which may or may not work with C# Wasm AOT.
C# Benchmarks
The source code of the C# benchmark executer can be found here. The way a C# benchmark should be defined is:
-
Create a C# class
SomeBenchmark
, which inherits fromBenchmark
. - Override some describing properties.
-
Override the
Parameters
property, which is of typeint[]
.SomeBenchmark
is initialized and executed with each of the values ofParameters
. This means that the performance of something can be evaulated with several parameter values. For example, ifParameters = new int[] { 1, 10, 100, 1000 }
andSomeBenchmark
sorts a list of numbers, then the analysis can be evaluated for a list of length 1, a list of length 10, a list of length 100 and a list of length 1000. -
Override the
public void Initialize(int parameter)
method. The value ofparameter
is one value from theParameters
array. This method is used to initialize stuff needed for the benchmark, but the performance is not evaluated. -
Override the
public object Execute()
method. This method should execute the code that is to be benchmarked. The method returns anobject
, which main purpose should be to ensure that the compiler does not remove the code within the method. The performance of this method is evaluated. -
Register
SomeBenchmark
under aBenchmarkCategory
inBenchmarking/Core/BenchmarkCategory.cs
. Note that you may need to define an extra category.
JavaScript Benchmarks
The source code of the JS benchmark executer can be found here. The way a JS benchmark should be defined is:
-
Create a JavaScript class
SomeBenchmark
. Note that the class name should be the same as the C# class name. - No need to override describing properties. The values from C# are dynamically injected wherever needed.
-
No need to define a
Parameters
array. The value from C# is dynamically injected wherever needed. -
Define the
function Initialize(parameter)
function. The behavior should be the same as the C# equivalent. This performance is not evaluated. -
Define the
function Execute()
function. The behavior should be the same as the C# equivalent. This performance is evaluated. - No need to register this class anywhere. It is dynamically injected wherever needed.
Benchmark Plots
Each benchmark is executed a certain number of times for each parameter value. The mean execution times of these runs represent data points, while the standard deviations represent error bars.
Benchmarks
The benchmarks.
ArrayBenchmarks
The benchmarks related to this category.
ArraySortInt
Generates an array of random ints.
Sorts the array.
The middle value of the sorted array.
The length of the array that is sorted.
ArraySortDouble
Generates an array of random doubles.
Sorts the array.
The middle value of the sorted array.
The length of the array that is sorted.
ArraySortIntQuick
Generates an array of random ints.
Sorts the array with a custom implemented Middle Point Pivot Quicksort (source).
The middle value of the sorted array.
The length of the array that is sorted.
ListBenchmarks
The benchmarks related to this category.
ListSortInt
Generates a list of random ints.
Sorts the list.
The middle value of the sorted list.
The length of the list.
ListSortDouble
Generates a list of random doubles.
Sorts the list.
The middle value of the sorted list.
The length of the list.
ListInsert
Generates a list of random ints.
Inserts 20% more values to the beginning of the list.
The middle value of the list.
The length of the list.
ListDelete
Generates a list of random ints.
Deletes 20% of the values from the beginning of the list.
The middle value of the list.
The length of the list.
MathBenchmarks
The benchmarks related to this category.
SummationInt
Generates a list of random ints.
Sums each number of the list.
The result of the summation.
The number of ints to sum.
SummationDouble
Generates a list of random doubles.
Sums each number of the list.
The result of the summation.
The number of doubles to sum.
MultiplicationInt
Generates a list of random ints.
Multiplies each number of the list.
The result of the multiplication.
The number of ints to multiply.
MultiplicationDouble
Generates a list of random doubles.
Multiplies each number of the list.
The result of the multiplication.
The number of doubles to multiply.
NewtonsMethodSecondDegree
Solves a root of a second degree function with a certain number of iterations.
The found root.
The number of iterations.
NewtonsMethodThirdDegree
Solves a root of a third degree function with a certain number of iterations.
The found root.
The number of iterations.
StringBenchmarks
The benchmarks related to this category.
StringConcatenation
Concatenates a single character to a string a certain number of times.
The middle character of the concatenated string.
The number of characters to concatenate.
StringConcatenationWithBuilder
Concatenates a single character to a string a certain number of times using a StringBuilder.
The middle character of the concatenated string.
The number of characters to concatenate.
DictionaryBenchmarks
The benchmarks related to this category.
DictionaryAccessInt
Generates a dictionary of random ints as keys with random ints as values.
Randomly accesses half of the keys and sums their respective values.
The sum of the values of the accessed keys.
The number of items in the dictionary.
DictionaryAccessString
Generates a dictionary of random strings as keys with random ints as values.
Randomly accesses half of the keys and sums their respective values.
The sum of the values of the accessed keys.
The number of items in the dictionary.
Uno Wasm Interpreted vs Blazor Wasm Interpreted
How does Uno.Wasm.Bootstrap compare to Blazor? Blazor does not yet support AOT and Blazor interpreted gives runtime memory errors for the full benchmarks. However, this section contains some limited benchmarks to compare Uno Wasm interpreted and Blazor Wasm interpreted.
ArrayBenchmarks
The benchmarks related to this category.
ArraySortInt
Generates an array of random ints.
Sorts the array.
The middle value of the sorted array.
The length of the array that is sorted.
ArraySortDouble
Generates an array of random doubles.
Sorts the array.
The middle value of the sorted array.
The length of the array that is sorted.
ArraySortIntQuick
Generates an array of random ints.
Sorts the array with a custom implemented Middle Point Pivot Quicksort (source).
The middle value of the sorted array.
The length of the array that is sorted.
ListBenchmarks
The benchmarks related to this category.
ListSortInt
Generates a list of random ints.
Sorts the list.
The middle value of the sorted list.
The length of the list.
ListSortDouble
Generates a list of random doubles.
Sorts the list.
The middle value of the sorted list.
The length of the list.
ListInsert
Generates a list of random ints.
Inserts 20% more values to the beginning of the list.
The middle value of the list.
The length of the list.
ListDelete
Generates a list of random ints.
Deletes 20% of the values from the beginning of the list.
The middle value of the list.
The length of the list.
MathBenchmarks
The benchmarks related to this category.
SummationInt
Generates a list of random ints.
Sums each number of the list.
The result of the summation.
The number of ints to sum.
SummationDouble
Generates a list of random doubles.
Sums each number of the list.
The result of the summation.
The number of doubles to sum.
MultiplicationInt
Generates a list of random ints.
Multiplies each number of the list.
The result of the multiplication.
The number of ints to multiply.
MultiplicationDouble
Generates a list of random doubles.
Multiplies each number of the list.
The result of the multiplication.
The number of doubles to multiply.
NewtonsMethodSecondDegree
Solves a root of a second degree function with a certain number of iterations.
The found root.
The number of iterations.
NewtonsMethodThirdDegree
Solves a root of a third degree function with a certain number of iterations.
The found root.
The number of iterations.
StringBenchmarks
The benchmarks related to this category.
StringConcatenation
Concatenates a single character to a string a certain number of times.
The middle character of the concatenated string.
The number of characters to concatenate.
StringConcatenationWithBuilder
Concatenates a single character to a string a certain number of times using a StringBuilder.
The middle character of the concatenated string.
The number of characters to concatenate.
DictionaryBenchmarks
The benchmarks related to this category.
DictionaryAccessInt
Generates a dictionary of random ints as keys with random ints as values.
Randomly accesses half of the keys and sums their respective values.
The sum of the values of the accessed keys.
The number of items in the dictionary.
DictionaryAccessString
Generates a dictionary of random strings as keys with random ints as values.
Randomly accesses half of the keys and sums their respective values.
The sum of the values of the accessed keys.
The number of items in the dictionary.