CSharpWasmBenchmark - Results

Update: Blazor Wasm AOT has now been released in ASP.NET Core 6 Preview 4. This benchmark uses that for AOT compilation.

CSharpWasmBenchmark is a project that benchmarks the performance of C# when compiled to Webassembly. The rationale behind this analysis is to find out whether it is viable to run performance critical applications written in C# in the browser and how the browser performance differs from "native" performance.

The performance of the following are measured: C# Runtime, C# Wasm AOT Blazor, C# Wasm Interpreted Blazor and JavaScript.

The C# Wasm compilation is still in a preview phase and improvements can probably be expected.

The JavaScript code is directly translated (as well as possible) from the C# source code, which may create certain differences between the implications of the code that is executed. You should always compare the source codes! The comparison between the performance of different C# execution targets are likely more reliable than that of comparison with JavaScript. JavaScript was included in this analysis since currently it is the standard frontend development programming language.

Note: The benchmarks may have errors in them and the C# and JavaScript source codes may not be equivalent. However, the C# code is always executed from the same source and should thus be comparable. Additionally, since C# Wasm AOT is still a somewhat experimental technology these results may not reflect the performance of the finalized product.

The JavaScript implementations of some of the sorting benchmarks are slightly incorrect, since JavaScript's sort does some strange string conversion. Maybe I'll update this sometime...

Unfortunately Blazor Wasm threw some memory error for larger parameter values. I do not currently have the time to investigate this and as such I just decreased the magnitude of the parameters. This caused certain C# runtime benchmarks to complete in "zero" time, or at least so fast that Stopwatch could not measure it. Do note that the executed code is just the same for C# runtime and both C# Blazor variants.

Conclusions

The performance of C# AOT compiled to Webassembly is often significantly better than that of C# Wasm Interpreted. Generally the improvement in the featured benchmarks lies within the range 2x - 5x better performance, but in some cases even a 20x performance benefit can be seen. Strangely, sorting doubles seems to be slower with AOT code.

C# Runtime can in some cases perform over 30x (even be faster in certain cases, but due to the "zero" time problem all values can not be calculated) faster than C# Wasm AOT. However, the performance is almost on par in some cases with C# Wasm AOT. Probably when no abstractions or framework calls are utilized. See for example the C# Wasm AOT performance difference between ArraySortInt (uses Array.Sort) and ArraySortIntQuick (uses a custom quicksort implementation). The performance of ArraySortIntQuick is about 20x better than that of ArraySortInt, even if they, in practice, do the same work. The performance difference between ArraySortInt and ArraySortIntQuick when executing under the C# Runtime is negligible.

The equivalent code executed as JavaScript is generally 15x faster than C# Wasm AOT. However, if the code avoids relying on framework methods and calls (for example ArraySortIntQuick, NewtonsMethodSecondDegree and NewtonsMethodThirdDegree) then the difference is almost negligible.

Webassembly promises, more or less, that it can enable "native" performance within a browser. C# often runs within a virtual machine and can maybe thus not achieve the full potential of Wasm. However, the difference in performance between C# Runtime and C# Wasm AOT varies from benchmark to benchmark and does generally not live up to the promise of Webassembly. With the benchmarks and assumptions of this project the conclusion is that C# Wasm AOT still has a long way to become a general and performant client side web programming platform. Hopefully this changes in the future.

Credits

This analysis was created by Acmion.

Contribute to CSharpWasmBenchmark in it's GitHub repository. Consider leaving a like if you found this analysis useful, interesting or informative.

Thanks to Uno.Wasm.Bootstrap, a project that provides an understandable way of AOT compiling C# to Wasm that just works and has minimal dependencies. Uno.Wasm.Bootstrap is no longer used in this benchmark, but for a long time it was used for C# Wasm AOT.

Benchmarking Strategy

This section describes how the benchmarks are created and executed. All benchmarks are contained within some categories, for example, there is an ArrayBenchmarks and a ListBenchmarks category. BenchmarkDotNet was not utilized in this project, because I wanted to ensure that the C# and JavaScript code and benchmarks are doing the same things. Additionally, BenchmarkDotNet uses some complex compilation strategies, which may or may not work with C# Wasm AOT.

C# Benchmarks

The source code of the C# benchmark executer can be found here. The way a C# benchmark should be defined is:

  1. Create a C# class SomeBenchmark, which inherits from Benchmark.
  2. Override some describing properties.
  3. Override the Parameters property, which is of type int[]. SomeBenchmark is initialized and executed with each of the values of Parameters. This means that the performance of something can be evaulated with several parameter values. For example, if Parameters = new int[] { 1, 10, 100, 1000 } and SomeBenchmark sorts a list of numbers, then the analysis can be evaluated for a list of length 1, a list of length 10, a list of length 100 and a list of length 1000.
  4. Override the public void Initialize(int parameter) method. The value of parameter is one value from the Parameters array. This method is used to initialize stuff needed for the benchmark, but the performance is not evaluated.
  5. Override the public object Execute() method. This method should execute the code that is to be benchmarked. The method returns an object, which main purpose should be to ensure that the compiler does not remove the code within the method. The performance of this method is evaluated.
  6. Register SomeBenchmark under a BenchmarkCategory in Benchmarking/Core/BenchmarkCategory.cs. Note that you may need to define an extra category.

JavaScript Benchmarks

The source code of the JS benchmark executer can be found here. The way a JS benchmark should be defined is:

  1. Create a JavaScript class SomeBenchmark. Note that the class name should be the same as the C# class name.
  2. No need to override describing properties. The values from C# are dynamically injected wherever needed.
  3. No need to define a Parameters array. The value from C# is dynamically injected wherever needed.
  4. Define the function Initialize(parameter) function. The behavior should be the same as the C# equivalent. This performance is not evaluated.
  5. Define the function Execute() function. The behavior should be the same as the C# equivalent. This performance is evaluated.
  6. No need to register this class anywhere. It is dynamically injected wherever needed.

Benchmark Plots

Each benchmark is executed a certain number of times for each parameter value. The mean execution times of these runs represent data points, while the standard deviations represent error bars.

Benchmarks

The benchmarks.

ArrayBenchmarks

The benchmarks related to this category.

ArraySortInt

Initialization Description:
Generates an array of random ints.
Benchmark Description:
Sorts the array.
Result Description:
The middle value of the sorted array.
Parameter Description:
The length of the array that is sorted.
Source Code:
C#, JavaScript
Performance:

ArraySortDouble

Initialization Description:
Generates an array of random doubles.
Benchmark Description:
Sorts the array.
Result Description:
The middle value of the sorted array.
Parameter Description:
The length of the array that is sorted.
Source Code:
C#, JavaScript
Performance:

ArraySortIntQuick

Initialization Description:
Generates an array of random ints.
Benchmark Description:
Sorts the array with a custom implemented Middle Point Pivot Quicksort (source).
Result Description:
The middle value of the sorted array.
Parameter Description:
The length of the array that is sorted.
Source Code:
C#, JavaScript
Performance:

ListBenchmarks

The benchmarks related to this category.

ListSortInt

Initialization Description:
Generates a list of random ints.
Benchmark Description:
Sorts the list.
Result Description:
The middle value of the sorted list.
Parameter Description:
The length of the list.
Source Code:
C#, JavaScript
Performance:

ListSortDouble

Initialization Description:
Generates a list of random doubles.
Benchmark Description:
Sorts the list.
Result Description:
The middle value of the sorted list.
Parameter Description:
The length of the list.
Source Code:
C#, JavaScript
Performance:

ListInsert

Initialization Description:
Generates a list of random ints.
Benchmark Description:
Inserts 20% more values to the beginning of the list.
Result Description:
The middle value of the list.
Parameter Description:
The length of the list.
Source Code:
C#, JavaScript
Performance:

ListDelete

Initialization Description:
Generates a list of random ints.
Benchmark Description:
Deletes 20% of the values from the beginning of the list.
Result Description:
The middle value of the list.
Parameter Description:
The length of the list.
Source Code:
C#, JavaScript
Performance:

MathBenchmarks

The benchmarks related to this category.

SummationInt

Initialization Description:
Generates a list of random ints.
Benchmark Description:
Sums each number of the list.
Result Description:
The result of the summation.
Parameter Description:
The number of ints to sum.
Source Code:
C#, JavaScript
Performance:

SummationDouble

Initialization Description:
Generates a list of random doubles.
Benchmark Description:
Sums each number of the list.
Result Description:
The result of the summation.
Parameter Description:
The number of doubles to sum.
Source Code:
C#, JavaScript
Performance:

MultiplicationInt

Initialization Description:
Generates a list of random ints.
Benchmark Description:
Multiplies each number of the list.
Result Description:
The result of the multiplication.
Parameter Description:
The number of ints to multiply.
Source Code:
C#, JavaScript
Performance:

MultiplicationDouble

Initialization Description:
Generates a list of random doubles.
Benchmark Description:
Multiplies each number of the list.
Result Description:
The result of the multiplication.
Parameter Description:
The number of doubles to multiply.
Source Code:
C#, JavaScript
Performance:

NewtonsMethodSecondDegree

Initialization Description:
Benchmark Description:
Solves a root of a second degree function with a certain number of iterations.
Result Description:
The found root.
Parameter Description:
The number of iterations.
Source Code:
C#, JavaScript
Performance:

NewtonsMethodThirdDegree

Initialization Description:
Benchmark Description:
Solves a root of a third degree function with a certain number of iterations.
Result Description:
The found root.
Parameter Description:
The number of iterations.
Source Code:
C#, JavaScript
Performance:

StringBenchmarks

The benchmarks related to this category.

StringConcatenation

Initialization Description:
Benchmark Description:
Concatenates a single character to a string a certain number of times.
Result Description:
The middle character of the concatenated string.
Parameter Description:
The number of characters to concatenate.
Source Code:
C#, JavaScript
Performance:

StringConcatenationWithBuilder

Initialization Description:
Benchmark Description:
Concatenates a single character to a string a certain number of times using a StringBuilder.
Result Description:
The middle character of the concatenated string.
Parameter Description:
The number of characters to concatenate.
Source Code:
C#, JavaScript
Performance:

DictionaryBenchmarks

The benchmarks related to this category.

DictionaryAccessInt

Initialization Description:
Generates a dictionary of random ints as keys with random ints as values.
Benchmark Description:
Randomly accesses half of the keys and sums their respective values.
Result Description:
The sum of the values of the accessed keys.
Parameter Description:
The number of items in the dictionary.
Source Code:
C#, JavaScript
Performance:

DictionaryAccessString

Initialization Description:
Generates a dictionary of random strings as keys with random ints as values.
Benchmark Description:
Randomly accesses half of the keys and sums their respective values.
Result Description:
The sum of the values of the accessed keys.
Parameter Description:
The number of items in the dictionary.
Source Code:
C#, JavaScript
Performance: