CSharpWasmBenchmark - Results

CSharpWasmBenchmark is a project that benchmarks the performance of C# when compiled to Webassembly. The rationale behind this analysis is to find out whether it is viable to run performance critical applications written in C# in the browser and how the browser performance differs from "native" performance.

The performance of the following are measured: C# Runtime, C# Wasm AOT Blazor, C# Wasm Interpreted Blazor, C Wasm and JavaScript. This C# code has been compiled with .NET 7.

Improvements can probably be expected to C# Wasm in the future.

The JavaScript code is directly translated (as well as possible) from the C# source code, which may create certain differences between the implications of the code that is executed. You should always compare the source codes! The comparison between the performance of different C# execution targets are likely more reliable than that of comparison with JavaScript. JavaScript was included in this analysis since currently it is the standard frontend development programming language.

Note: The benchmarks may have errors in them and the source codes may not be equivalent. However, the C# code is always executed from the same source and should thus be comparable. Additionally, since C# Wasm AOT and Wasm itself are still somewhat experimental technologies, these results may not reflect the performance of the finalized product.

Unfortunately Blazor Wasm threw some memory error for larger parameter values. I do not currently have the time to investigate this and as such I just decreased the magnitude of the parameters. Note that the executed code is just the same for C# runtime and both C# Blazor variants.

Conclusions

Platform Total Time Spent in Benchmarks (ms) Relative Performance
Loading...

Table 1: The total time spent in all the benchmarks. This data can be interpreted to give some sort of an average estimate of how well the different platforms perform. However, the performance depends on the parameter values. See the figures below for a more detailed overview.

The performance of C# AOT compiled to Webassembly is often significantly better than that of C# Wasm Interpreted. Table 1 shows the total time spent in benchmarking each platform. Running the benchmarks in C# Wasm AOT takes approximately 2.69x longer than in the C# runtime and running the benchmarks in C# Wasm Interpreted takes approximately 33.19x longer than in the C# runtime. These values are rough estimates and depend on parameter values. See more details in the figures below.

Generally the performance improvement of AOT in the featured benchmarks lies within the range 2x - 5x, but in some cases even a 20x performance benefit can be seen. Strangely, for example, sorting doubles seems to be slightly slower with AOT code.

C# Runtime can in some cases perform over 30x (even faster in certain cases) faster than C# Wasm AOT. However, the performance is almost on par in some cases with C# Wasm AOT. Probably when no abstractions or framework calls are utilized. See for example the C# Wasm AOT performance difference between ArraySortInt (uses Array.Sort) and ArraySortIntQuick (uses a custom quicksort implementation). The performance of ArraySortIntQuick in C# Wasm AOT is about 20x better than that of ArraySortInt, even if they, in practice, do the same work. The performance difference between ArraySortInt and ArraySortIntQuick when executing under the C# Runtime is negligible.

The equivalent code executed as JavaScript is generally faster than C# Wasm AOT. However, if the code avoids relying on framework methods and calls (for example ArraySortIntQuick, NewtonsMethodSecondDegree and NewtonsMethodThirdDegree) then the difference is almost negligible.

Webassembly promises, more or less, that it can enable "native" performance within a browser. C# often runs within a virtual machine and can maybe thus not achieve the full potential of Wasm. However, the difference in performance between C# Runtime and C# Wasm AOT varies from benchmark to benchmark and does generally not live up to the promise of Webassembly. Even JS is often faster than C# Wasm AOT. With the benchmarks and assumptions of this project the conclusion is that C# Wasm AOT still has a long way to become a general and performant client side web programming platform. Hopefully this changes in the future.

Credits

This analysis initially created by Acmion.

Contribute to CSharpWasmBenchmark in it's GitHub repository. Consider leaving a like if you found this analysis useful, interesting or informative.

Thanks to gabrielgt for his contributions in updating the project to .NET 7 and the C Wasm implementation.

Thanks to Uno.Wasm.Bootstrap, a project that provides an understandable way of AOT compiling C# to Wasm that just works and has minimal dependencies. Uno.Wasm.Bootstrap is no longer used in this benchmark, but for a long time it was used for C# Wasm AOT.

Benchmarking Strategy

This section describes how the benchmarks are created and executed. All benchmarks are contained within some categories, for example, there is an ArrayBenchmarks and a ListBenchmarks category. BenchmarkDotNet was not utilized in this project, because I wanted to ensure that the C# and JavaScript code and benchmarks are doing the same things. Additionally, BenchmarkDotNet uses some complex compilation strategies, which may or may not work with C# Wasm AOT.

C# Benchmarks

The source code of the C# benchmark executer can be found here. The way a C# benchmark should be defined is:

  1. Create a C# class SomeBenchmark, which inherits from Benchmark.
  2. Override some describing properties.
  3. Override the Parameters property, which is of type int[]. SomeBenchmark is initialized and executed with each of the values of Parameters. This means that the performance of something can be evaulated with several parameter values. For example, if Parameters = new int[] { 1, 10, 100, 1000 } and SomeBenchmark sorts a list of numbers, then the analysis can be evaluated for a list of length 1, a list of length 10, a list of length 100 and a list of length 1000.
  4. Override the public void Initialize(int parameter) method. The value of parameter is one value from the Parameters array. This method is used to initialize stuff needed for the benchmark, but the performance is not evaluated.
  5. Override the public object Execute() method. This method should execute the code that is to be benchmarked. The method returns an object, which main purpose should be to ensure that the compiler does not remove the code within the method. The performance of this method is evaluated.
  6. Register SomeBenchmark under a BenchmarkCategory in Benchmarking/Core/BenchmarkCategory.cs. Note that you may need to define an extra category.

JavaScript Benchmarks

The source code of the JS benchmark executer can be found here. The way a JS benchmark should be defined is:

  1. Create a JavaScript class SomeBenchmark. Note that the class name should be the same as the C# class name.
  2. No need to override describing properties. The values from C# are dynamically injected wherever needed.
  3. No need to define a Parameters array. The value from C# is dynamically injected wherever needed.
  4. Define the function Initialize(parameter) function. The behavior should be the same as the C# equivalent. This performance is not evaluated.
  5. Define the function Execute() function. The behavior should be the same as the C# equivalent. This performance is evaluated.
  6. No need to register this class anywhere. It is dynamically injected wherever needed.

Benchmark Plots

Each benchmark is executed a certain number of times for each parameter value. The mean execution times of these runs represent data points, while the standard deviations represent error bars.

Benchmarks

The benchmarks.

ArrayBenchmarks

The benchmarks related to this category.

ArraySortInt

Initialization Description:
Generates an array of random ints.
Benchmark Description:
Sorts the array.
Result Description:
The middle value of the sorted array.
Parameter Description:
The length of the array that is sorted.
Source Code:
C#, JavaScript, C
Performance:
2k4k6k8k10k02468
C Wasm (Edge 111.0.1661) CS Runtime CS Wasm AOT Blazor (Edge 111.0.1661) CS Wasm Interpreted Blazor (Edge 111.0.1661) JavaScript (Edge 111.0.1661)Parameter ValueElapsed Milliseconds

ArraySortDouble

Initialization Description:
Generates an array of random doubles.
Benchmark Description:
Sorts the array.
Result Description:
The middle value of the sorted array.
Parameter Description:
The length of the array that is sorted.
Source Code:
C#, JavaScript, C
Performance:
2k4k6k8k10k0246810
C Wasm (Edge 111.0.1661) CS Runtime CS Wasm AOT Blazor (Edge 111.0.1661) CS Wasm Interpreted Blazor (Edge 111.0.1661) JavaScript (Edge 111.0.1661)Parameter ValueElapsed Milliseconds

ArraySortIntQuick

Initialization Description:
Generates an array of random ints.
Benchmark Description:
Sorts the array with a custom implemented Middle Point Pivot Quicksort (source).
Result Description:
The middle value of the sorted array.
Parameter Description:
The length of the array that is sorted.
Source Code:
C#, JavaScript, C
Performance:
2k4k6k8k10k01234
C Wasm (Edge 111.0.1661) CS Runtime CS Wasm AOT Blazor (Edge 111.0.1661) CS Wasm Interpreted Blazor (Edge 111.0.1661) JavaScript (Edge 111.0.1661)Parameter ValueElapsed Milliseconds

ListBenchmarks

The benchmarks related to this category.

ListSortInt

Initialization Description:
Generates a list of random ints.
Benchmark Description:
Sorts the list.
Result Description:
The middle value of the sorted list.
Parameter Description:
The length of the list.
Source Code:
C#, JavaScript, C
Performance:

ListSortDouble

Initialization Description:
Generates a list of random doubles.
Benchmark Description:
Sorts the list.
Result Description:
The middle value of the sorted list.
Parameter Description:
The length of the list.
Source Code:
C#, JavaScript, C
Performance:

ListInsert

Initialization Description:
Generates a list of random ints.
Benchmark Description:
Inserts 20% more values to the beginning of the list.
Result Description:
The middle value of the list.
Parameter Description:
The length of the list.
Source Code:
C#, JavaScript, C
Performance:

ListDelete

Initialization Description:
Generates a list of random ints.
Benchmark Description:
Deletes 20% of the values from the beginning of the list.
Result Description:
The middle value of the list.
Parameter Description:
The length of the list.
Source Code:
C#, JavaScript, C
Performance:

MathBenchmarks

The benchmarks related to this category.

SummationInt

Initialization Description:
Generates a list of random ints.
Benchmark Description:
Sums each number of the list.
Result Description:
The result of the summation.
Parameter Description:
The number of ints to sum.
Source Code:
C#, JavaScript, C
Performance:

SummationDouble

Initialization Description:
Generates a list of random doubles.
Benchmark Description:
Sums each number of the list.
Result Description:
The result of the summation.
Parameter Description:
The number of doubles to sum.
Source Code:
C#, JavaScript, C
Performance:

MultiplicationInt

Initialization Description:
Generates a list of random ints.
Benchmark Description:
Multiplies each number of the list.
Result Description:
The result of the multiplication.
Parameter Description:
The number of ints to multiply.
Source Code:
C#, JavaScript, C
Performance:

MultiplicationDouble

Initialization Description:
Generates a list of random doubles.
Benchmark Description:
Multiplies each number of the list.
Result Description:
The result of the multiplication.
Parameter Description:
The number of doubles to multiply.
Source Code:
C#, JavaScript, C
Performance:

NewtonsMethodSecondDegree

Initialization Description:
Benchmark Description:
Solves a root of a second degree function with a certain number of iterations.
Result Description:
The found root.
Parameter Description:
The number of iterations.
Source Code:
C#, JavaScript, C
Performance:

NewtonsMethodThirdDegree

Initialization Description:
Benchmark Description:
Solves a root of a third degree function with a certain number of iterations.
Result Description:
The found root.
Parameter Description:
The number of iterations.
Source Code:
C#, JavaScript, C
Performance:

StringBenchmarks

The benchmarks related to this category.

StringConcatenation

Initialization Description:
Benchmark Description:
Concatenates a single character to a string a certain number of times.
Result Description:
The middle character of the concatenated string.
Parameter Description:
The number of characters to concatenate.
Source Code:
C#, JavaScript, C
Performance:
Run the benchmarks to inspect the performance.

StringConcatenationWithBuilder

Initialization Description:
Benchmark Description:
Concatenates a single character to a string a certain number of times using a StringBuilder.
Result Description:
The middle character of the concatenated string.
Parameter Description:
The number of characters to concatenate.
Source Code:
C#, JavaScript, C
Performance:
Run the benchmarks to inspect the performance.

DictionaryBenchmarks

The benchmarks related to this category.

DictionaryAccessInt

Initialization Description:
Generates a dictionary of random ints as keys with random ints as values.
Benchmark Description:
Randomly accesses half of the keys and sums their respective values.
Result Description:
The sum of the values of the accessed keys.
Parameter Description:
The number of items in the dictionary.
Source Code:
C#, JavaScript, C
Performance:
2k4k6k8k10k00.20.40.60.8
C Wasm (Edge 111.0.1661) CS Runtime CS Wasm AOT Blazor (Edge 111.0.1661)Parameter ValueElapsed Milliseconds

DictionaryAccessString

Initialization Description:
Generates a dictionary of random strings as keys with random ints as values.
Benchmark Description:
Randomly accesses half of the keys and sums their respective values.
Result Description:
The sum of the values of the accessed keys.
Parameter Description:
The number of items in the dictionary.
Source Code:
C#, JavaScript, C
Performance:
2k4k6k8k10k00.511.5
C Wasm (Edge 111.0.1661) CS Runtime CS Wasm AOT Blazor (Edge 111.0.1661)Parameter ValueElapsed Milliseconds