ruby.git/benchmark, branch v3_4_9

[DOC] Improve formatting in Markdown files (#12322)

2024-12-13T01:49:45+00:00

* Fix WASM bullet/code indentation

* Use `console` code highlighting where appropriate

… which handles the prefix `$` correctly.

* Migrate feature proposal template to MarkDown

* Set language on code blocks

[ruby/strscan] Add scan and search benchmark

2024-11-27T00:24:06+00:00

(https://github.com/ruby/strscan/pull/111)

# Why?
To improve the parsing process, I would like to add benchmarks for all
parsing processes.

## scan
- scan_full(regexp, false, true) == StringScanner#check
- scan_full(regexp, false, false) ==  StringScanner#match?

### CRuby

```
$ benchmark-driver benchmark/scan.yaml
Warming up --------------------------------------
          check(reg)    10.558M i/s -     10.848M times in 1.027445s (94.71ns/i)
          check(str)    13.368M i/s -     13.782M times in 1.030978s (74.80ns/i)
         match?(reg)    16.080M i/s -     16.247M times in 1.010340s (62.19ns/i)
         match?(str)    23.336M i/s -     23.501M times in 1.007088s (42.85ns/i)
Calculating -------------------------------------
          check(reg)    11.601M i/s -     31.675M times in 2.730287s (86.20ns/i)
          check(str)    15.217M i/s -     40.104M times in 2.635475s (65.72ns/i)
         match?(reg)    18.781M i/s -     48.241M times in 2.568662s (53.25ns/i)
         match?(str)    29.441M i/s -     70.007M times in 2.377840s (33.97ns/i)

Comparison:
         match?(str):  29441324.5 i/s
         match?(reg):  18780543.7 i/s - 1.57x  slower
          check(str):  15217130.1 i/s - 1.93x  slower
          check(reg):  11601371.2 i/s - 2.54x  slower
```
### JRuby

```
$ benchmark-driver benchmark/scan.yaml
Warming up --------------------------------------
          check(reg)     8.129M i/s -      8.090M times in 0.995222s (123.02ns/i)
          check(str)    16.691M i/s -     16.616M times in 0.995519s (59.91ns/i)
         match?(reg)     8.979M i/s -      9.001M times in 1.002440s (111.37ns/i)
         match?(str)    26.138M i/s -     26.011M times in 0.995150s (38.26ns/i)
Calculating -------------------------------------
          check(reg)    11.808M i/s -     24.387M times in 2.065238s (84.69ns/i)
          check(str)    31.762M i/s -     50.072M times in 1.576495s (31.48ns/i)
         match?(reg)    13.944M i/s -     26.936M times in 1.931719s (71.71ns/i)
         match?(str)    50.872M i/s -     78.414M times in 1.541392s (19.66ns/i)

Comparison:
         match?(str):  50872250.2 i/s
          check(str):  31761544.3 i/s - 1.60x  slower
         match?(reg):  13944219.6 i/s - 3.65x  slower
          check(reg):  11808244.1 i/s - 4.31x  slower
```

## search
- search_full(regexp, false, true) == StringScanner#check_until
- search_full(regexp, false, false) == StringScanner#exist?
```
$ benchmark-driver benchmark/search.yaml
Warming up --------------------------------------
    check_until(reg)     9.338M i/s -      9.456M times in 1.012573s (107.09ns/i)
    check_until(str)    11.385M i/s -     11.979M times in 1.052173s (87.83ns/i)
         exist?(reg)    13.416M i/s -     13.517M times in 1.007532s (74.54ns/i)
         exist?(str)    17.976M i/s -     18.677M times in 1.038981s (55.63ns/i)
Calculating -------------------------------------
    check_until(reg)    10.297M i/s -     28.015M times in 2.720634s (97.11ns/i)
    check_until(str)    12.684M i/s -     34.156M times in 2.692853s (78.84ns/i)
         exist?(reg)    15.184M i/s -     40.249M times in 2.650786s (65.86ns/i)
         exist?(str)    21.426M i/s -     53.928M times in 2.517008s (46.67ns/i)

Comparison:
         exist?(str):  21425527.1 i/s
         exist?(reg):  15183679.9 i/s - 1.41x  slower
    check_until(str):  12684053.7 i/s - 1.69x  slower
    check_until(reg):  10297134.8 i/s - 2.08x  slower
```

### JRuby
```
$ benchmark-driver benchmark/search.yaml
Warming up --------------------------------------
    check_until(reg)     7.646M i/s -      7.649M times in 1.000381s (130.78ns/i)
    check_until(str)    13.075M i/s -     13.010M times in 0.995048s (76.48ns/i)
         exist?(reg)     8.728M i/s -      8.684M times in 0.994921s (114.57ns/i)
         exist?(str)    20.609M i/s -     20.514M times in 0.995399s (48.52ns/i)
Calculating -------------------------------------
    check_until(reg)     9.371M i/s -     22.939M times in 2.447900s (106.71ns/i)
    check_until(str)    22.760M i/s -     39.225M times in 1.723414s (43.94ns/i)
         exist?(reg)    11.758M i/s -     26.185M times in 2.226997s (85.05ns/i)
         exist?(str)    34.564M i/s -     61.827M times in 1.788749s (28.93ns/i)

Comparison:
         exist?(str):  34564306.2 i/s
    check_until(str):  22759878.4 i/s - 1.52x  slower
         exist?(reg):  11757927.4 i/s - 2.94x  slower
    check_until(reg):   9371009.3 i/s - 3.69x  slower
```

https://github.com/ruby/strscan/commit/81a80a176b

Get rid of JSON benchmarks

2024-11-05T11:19:55+00:00

Fixup b1fc1af4440. Removed benchmark files from ruby/json

2024-11-05T02:02:13+00:00

Elide Generator::State allocation until a `to_json` method has to be called

2024-11-01T04:04:24+00:00

Fix: https://github.com/ruby/json/issues/655

For very small documents, the biggest performance gap with alternatives is
that the API impose that we allocate the `State` object. In a real world app
this doesn't make much of a difference, but when running in a micro-benchmark
this doubles the allocations, causing twice the amount of GC runs, making us
look bad.

However, unless we have to call a `to_json` method, the `State` object isn't
visible, so with some refactoring, we can elude that allocation entirely.

Instead we allocate the State internal struct on the stack, and if we need
to call a `to_json` method, we allocate the `State` and spill the struct on
the heap.

As a result, `JSON.generate` is now as fast as re-using a `State` instance,
as long as only primitives are generated.

Before:
```
== Encoding small mixed (34 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
        json (reuse)   598.654k i/100ms
                json   400.542k i/100ms
                  oj   533.353k i/100ms
Calculating -------------------------------------
        json (reuse)      6.371M (± 8.6%) i/s  (156.96 ns/i) -     31.729M in   5.059195s
                json      4.120M (± 6.6%) i/s  (242.72 ns/i) -     20.828M in   5.090549s
                  oj      5.622M (± 6.4%) i/s  (177.86 ns/i) -     28.268M in   5.061473s

Comparison:
        json (reuse):  6371126.6 i/s
                  oj:  5622452.0 i/s - same-ish: difference falls within error
                json:  4119991.1 i/s - 1.55x  slower

== Encoding small nested array (121 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
        json (reuse)   248.125k i/100ms
                json   215.255k i/100ms
                  oj   217.531k i/100ms
Calculating -------------------------------------
        json (reuse)      2.628M (± 6.1%) i/s  (380.55 ns/i) -     13.151M in   5.030281s
                json      2.185M (± 6.7%) i/s  (457.74 ns/i) -     10.978M in   5.057655s
                  oj      2.217M (± 6.7%) i/s  (451.10 ns/i) -     11.094M in   5.044844s

Comparison:
        json (reuse):  2627799.4 i/s
                  oj:  2216824.8 i/s - 1.19x  slower
                json:  2184669.5 i/s - 1.20x  slower

== Encoding small hash (65 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
        json (reuse)   641.334k i/100ms
                json   322.745k i/100ms
                  oj   642.450k i/100ms
Calculating -------------------------------------
        json (reuse)      7.133M (± 6.5%) i/s  (140.19 ns/i) -     35.915M in   5.068201s
                json      4.615M (± 7.0%) i/s  (216.70 ns/i) -     22.915M in   5.003718s
                  oj      6.912M (± 6.4%) i/s  (144.68 ns/i) -     34.692M in   5.047690s

Comparison:
        json (reuse):  7133123.3 i/s
                  oj:  6911977.1 i/s - same-ish: difference falls within error
                json:  4614696.6 i/s - 1.55x  slower
```

After:

```
== Encoding small mixed (34 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
        json (reuse)   572.751k i/100ms
                json   457.741k i/100ms
                  oj   512.247k i/100ms
Calculating -------------------------------------
        json (reuse)      6.324M (± 6.9%) i/s  (158.12 ns/i) -     31.501M in   5.023093s
                json      6.263M (± 6.9%) i/s  (159.66 ns/i) -     31.126M in   5.017086s
                  oj      5.569M (± 6.6%) i/s  (179.56 ns/i) -     27.661M in   5.003739s

Comparison:
        json (reuse):  6324183.5 i/s
                json:  6263204.9 i/s - same-ish: difference falls within error
                  oj:  5569049.2 i/s - same-ish: difference falls within error

== Encoding small nested array (121 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
        json (reuse)   258.505k i/100ms
                json   242.335k i/100ms
                  oj   220.678k i/100ms
Calculating -------------------------------------
        json (reuse)      2.589M (± 9.6%) i/s  (386.17 ns/i) -     12.925M in   5.071853s
                json      2.594M (± 6.6%) i/s  (385.46 ns/i) -     13.086M in   5.083035s
                  oj      2.250M (± 2.3%) i/s  (444.43 ns/i) -     11.255M in   5.004707s

Comparison:
        json (reuse):  2589499.6 i/s
                json:  2594321.0 i/s - same-ish: difference falls within error
                  oj:  2250064.0 i/s - 1.15x  slower

== Encoding small hash (65 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
        json (reuse)   656.373k i/100ms
                json   644.135k i/100ms
                  oj   650.283k i/100ms
Calculating -------------------------------------
        json (reuse)      7.202M (± 7.1%) i/s  (138.84 ns/i) -     36.101M in   5.051438s
                json      7.278M (± 1.7%) i/s  (137.40 ns/i) -     36.716M in   5.046300s
                  oj      7.036M (± 1.7%) i/s  (142.12 ns/i) -     35.766M in   5.084729s

Comparison:
        json (reuse):  7202447.9 i/s
                json:  7277883.0 i/s - same-ish: difference falls within error
                  oj:  7036115.2 i/s - same-ish: difference falls within error

```

[ruby/json] Use JSON.generate instead of JSON.dump for benchmarking

2024-11-01T04:04:24+00:00

https://github.com/ruby/json/commit/97b61edce1

[ruby/json] Optimize `fbuffer_append_long`

2024-10-29T04:25:01+00:00

Ref: https://github.com/ruby/json/issues/655

Rather than to write the number backward, and then reverse
the buffer, we can start from the back of the buffer and write
the number in the proper direction.

Before:

```
== Encoding integers (8009 bytes)
ruby 3.3.4 (2024-07-09 revision https://github.com/ruby/json/commit/be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
                json     8.606k i/100ms
                  oj     9.598k i/100ms
Calculating -------------------------------------
                json     86.059k (± 0.8%) i/s   (11.62 μs/i) -    430.300k in   5.000416s
                  oj     97.409k (± 0.6%) i/s   (10.27 μs/i) -    489.498k in   5.025360s

Comparison:
                json:    86058.8 i/s
                  oj:    97408.8 i/s - 1.13x  faster
```

After:

```
== Encoding integers (8009 bytes)
ruby 3.3.4 (2024-07-09 revision https://github.com/ruby/json/commit/be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
        json (reuse)     9.500k i/100ms
                json     9.359k i/100ms
                  oj     9.722k i/100ms
Calculating -------------------------------------
        json (reuse)     96.270k (± 0.4%) i/s   (10.39 μs/i) -    484.500k in   5.032777s
                json     94.800k (± 2.2%) i/s   (10.55 μs/i) -    477.309k in   5.037495s
                  oj     97.131k (± 0.7%) i/s   (10.30 μs/i) -    486.100k in   5.004822s

Comparison:
        json (reuse):    96270.1 i/s
                  oj:    97130.5 i/s - same-ish: difference falls within error
                json:    94799.9 i/s - same-ish: difference falls within error
```

https://github.com/ruby/json/commit/0655b58d14

[ruby/json] Encoding benchmark updates

2024-10-26T09:44:15+00:00

Remove `rapidjson` as it's 2x slower most benchmarks, and on
par on a couple of them, so it's not telling us much here.

Configure `Oj` in compat mode so it generate the same JSON
on the `many to_json` benchmark.

```
== Encoding small nested array (121 bytes)
ruby 3.4.0preview2 (2024-10-07 master https://github.com/ruby/json/commit/32c733f57b) +YJIT +PRISM [arm64-darwin23]
Warming up --------------------------------------
        json (reuse)   220.202k i/100ms
                json   162.190k i/100ms
                  oj   222.094k i/100ms
Calculating -------------------------------------
        json (reuse)      2.322M (± 1.3%) i/s  (430.72 ns/i) -     11.671M in   5.027655s
                json      1.707M (± 1.2%) i/s  (585.76 ns/i) -      8.596M in   5.035996s
                  oj      2.248M (± 1.4%) i/s  (444.94 ns/i) -     11.327M in   5.040712s

Comparison:
        json (reuse):  2321686.9 i/s
                  oj:  2247509.6 i/s - 1.03x  slower
                json:  1707179.3 i/s - 1.36x  slower

== Encoding small hash (65 bytes)
ruby 3.4.0preview2 (2024-10-07 master https://github.com/ruby/json/commit/32c733f57b) +YJIT +PRISM [arm64-darwin23]
Warming up --------------------------------------
        json (reuse)   446.184k i/100ms
                json   265.594k i/100ms
                  oj   653.226k i/100ms
Calculating -------------------------------------
        json (reuse)      4.980M (± 1.4%) i/s  (200.82 ns/i) -     24.986M in   5.018729s
                json      2.763M (± 1.8%) i/s  (361.94 ns/i) -     13.811M in   5.000434s
                  oj      7.232M (± 1.4%) i/s  (138.28 ns/i) -     36.581M in   5.059377s

Comparison:
        json (reuse):  4979642.4 i/s
                  oj:  7231624.4 i/s - 1.45x  faster
                json:  2762890.1 i/s - 1.80x  slower

== Encoding mixed utf8 (5003001 bytes)
ruby 3.4.0preview2 (2024-10-07 master https://github.com/ruby/json/commit/32c733f57b) +YJIT +PRISM [arm64-darwin23]
Warming up --------------------------------------
                json    34.000 i/100ms
                  oj    36.000 i/100ms
Calculating -------------------------------------
                json    357.772 (± 4.8%) i/s    (2.80 ms/i) -      1.802k in   5.047308s
                  oj    327.521 (± 1.5%) i/s    (3.05 ms/i) -      1.656k in   5.057241s

Comparison:
                json:      357.8 i/s
                  oj:      327.5 i/s - 1.09x  slower

== Encoding mostly utf8 (5001001 bytes)
ruby 3.4.0preview2 (2024-10-07 master https://github.com/ruby/json/commit/32c733f57b) +YJIT +PRISM [arm64-darwin23]
Warming up --------------------------------------
                json    26.000 i/100ms
                  oj    36.000 i/100ms
Calculating -------------------------------------
                json    294.357 (±10.5%) i/s    (3.40 ms/i) -      1.456k in   5.028862s
                  oj    352.826 (± 8.2%) i/s    (2.83 ms/i) -      1.764k in   5.045651s

Comparison:
                json:      294.4 i/s
                  oj:      352.8 i/s - same-ish: difference falls within error

== Encoding twitter.json (466906 bytes)
ruby 3.4.0preview2 (2024-10-07 master https://github.com/ruby/json/commit/32c733f57b) +YJIT +PRISM [arm64-darwin23]
Warming up --------------------------------------
                json   206.000 i/100ms
                  oj   229.000 i/100ms
Calculating -------------------------------------
                json      2.064k (± 9.3%) i/s  (484.55 μs/i) -     10.300k in   5.056409s
                  oj      2.121k (± 8.4%) i/s  (471.47 μs/i) -     10.534k in   5.012315s

Comparison:
                json:     2063.8 i/s
                  oj:     2121.0 i/s - same-ish: difference falls within error

== Encoding citm_catalog.json (500298 bytes)
ruby 3.4.0preview2 (2024-10-07 master https://github.com/ruby/json/commit/32c733f57b) +YJIT +PRISM [arm64-darwin23]
Warming up --------------------------------------
                json   119.000 i/100ms
                  oj   126.000 i/100ms
Calculating -------------------------------------
                json      1.317k (± 2.3%) i/s  (759.18 μs/i) -      6.664k in   5.061781s
                  oj      1.261k (± 2.9%) i/s  (793.11 μs/i) -      6.300k in   5.000714s

Comparison:
                json:     1317.2 i/s
                  oj:     1260.9 i/s - same-ish: difference falls within error

== Encoding canada.json (2090234 bytes)
ruby 3.4.0preview2 (2024-10-07 master https://github.com/ruby/json/commit/32c733f57b) +YJIT +PRISM [arm64-darwin23]
Warming up --------------------------------------
                json     1.000 i/100ms
                  oj     1.000 i/100ms
Calculating -------------------------------------
                json     19.590 (± 0.0%) i/s   (51.05 ms/i) -     98.000 in   5.004485s
                  oj     19.003 (± 0.0%) i/s   (52.62 ms/i) -     95.000 in   5.002276s

Comparison:
                json:       19.6 i/s
                  oj:       19.0 i/s - 1.03x  slower

== Encoding many #to_json calls (2701 bytes)
ruby 3.4.0preview2 (2024-10-07 master https://github.com/ruby/json/commit/32c733f57b) +YJIT +PRISM [arm64-darwin23]
Warming up --------------------------------------
                json     2.556k i/100ms
                  oj     2.332k i/100ms
Calculating -------------------------------------
                json     25.367k (± 1.7%) i/s   (39.42 μs/i) -    127.800k in   5.039438s
                  oj     23.743k (± 1.5%) i/s   (42.12 μs/i) -    118.932k in   5.010303s

Comparison:
                json:    25367.3 i/s
                  oj:    23743.3 i/s - 1.07x  slower

```

https://github.com/ruby/json/commit/5a64fd5b6f

[ruby/json] Reduce encoding benchmark size

2024-10-26T09:44:15+00:00

Profiling revealed that we were spending lots of time growing the buffer.
Buffer operations is definitely something we want to optimize, but for
this specific benchmark what we're interested in is UTF-8 scanning performance.

Each iteration of the two scaning benchmark were producing 20MB of JSON,
now they only produce 5MB.

Now:

```
== Encoding mostly utf8 (5001001 bytes)
ruby 3.4.0dev (2024-10-18T19:01:45Z master https://github.com/ruby/json/commit/7be9a333ca) +YJIT +PRISM [arm64-darwin23]
Warming up --------------------------------------
                json    35.000 i/100ms
                  oj    36.000 i/100ms
           rapidjson    10.000 i/100ms
Calculating -------------------------------------
                json    359.161 (± 1.4%) i/s    (2.78 ms/i) -      1.820k in   5.068542s
                  oj    359.699 (± 0.6%) i/s    (2.78 ms/i) -      1.800k in   5.004291s
           rapidjson     99.687 (± 2.0%) i/s   (10.03 ms/i) -    500.000 in   5.017321s

Comparison:
                json:      359.2 i/s
                  oj:      359.7 i/s - same-ish: difference falls within error
           rapidjson:       99.7 i/s - 3.60x  slower
```

https://github.com/ruby/json/commit/1a338532d2

[ruby/json] convert_UTF8_to_JSON: repurpose the escape tables into size tables

2024-10-26T09:44:15+00:00

Since we're looking up the table anyway, we might as well store the
UTF-8 char length in it. For single byte characters that don't need
escaping we store `0`.

This helps on strings with lots of multi-byte characters:

Before:

```
== Encoding mostly utf8 (20004001 bytes)
ruby 3.3.4 (2024-07-09 revision https://github.com/ruby/json/commit/be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
                json     6.000 i/100ms
                  oj    10.000 i/100ms
           rapidjson     2.000 i/100ms
Calculating -------------------------------------
                json     67.978 (± 1.5%) i/s   (14.71 ms/i) -    342.000 in   5.033062s
                  oj    100.876 (± 2.0%) i/s    (9.91 ms/i) -    510.000 in   5.058080s
           rapidjson     26.389 (± 7.6%) i/s   (37.89 ms/i) -    132.000 in   5.027681s

Comparison:
                json:       68.0 i/s
                  oj:      100.9 i/s - 1.48x  faster
           rapidjson:       26.4 i/s - 2.58x  slower
```

After:

```
== Encoding mostly utf8 (20004001 bytes)
ruby 3.3.4 (2024-07-09 revision https://github.com/ruby/json/commit/be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
                json     7.000 i/100ms
                  oj    10.000 i/100ms
           rapidjson     2.000 i/100ms
Calculating -------------------------------------
                json     75.187 (± 2.7%) i/s   (13.30 ms/i) -    378.000 in   5.030111s
                  oj     95.196 (± 2.1%) i/s   (10.50 ms/i) -    480.000 in   5.043565s
           rapidjson     25.969 (± 3.9%) i/s   (38.51 ms/i) -    130.000 in   5.011471s

Comparison:
                json:       75.2 i/s
                  oj:       95.2 i/s - 1.27x  faster
           rapidjson:       26.0 i/s - 2.90x  slower
```

https://github.com/ruby/json/commit/51e2631d1f