summaryrefslogtreecommitdiff
path: root/tool/transform_mjit_header.rb
AgeCommit message (Collapse)Author
2018-11-05Fix a typo [ci skip]kazu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65546 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-05Create MJIT header with strict permissionnobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65544 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-19transform_mjit_header.rb: fix up r65169nobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65173 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-19transform_mjit_header.rb: `inline` may be a macronobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65170 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-19transform_mjit_header.rb: add system_header pragmanobu
* tool/transform_mjit_header.rb: insert GCC system_header pragma to get rid of errors caused by the contents included from system headers. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65169 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-09-26revert r64847, r64846 and r64839k0kubun
because r64849 seems to fix issues which we were confused about. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64850 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-09-26revert r64838 and r64839k0kubun
because some build failures persisted git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64846 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-09-25Revert "Revert r64824 to fix build failure on AppVeyor"k0kubun
This reverts commit r64829. I'll prepare another temporary fix, but I'll separately commit that to make it easier to revert that later. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64838 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-09-25Revert r64824 to fix build failure on AppVeyork0kubun
AppVeyor msys2/MinGW build started to fail like: https://ci.appveyor.com/project/ruby/ruby/build/9722/job/b94kixi004klmye3 Until I can investigate that, I revert this for now. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64829 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-09-24insns.def: optimize & and | of Integer [experimental]k0kubun
not optimizing Array#& and Array#| because vm_insnhelper.c can't easily inline it (large amount of array.c code would be needed in vm_insnhelper.c) and the method body is a little complicated compared to Integer's ones. So I thought only Integer#& and Integer#| have a significant impact, and eliminating unnecessary branches would contribute to JIT's performance. vm_insnhelper.c: ditto tool/transform_mjit_header.rb: make sure these instructions are inlined on JIT. compile.c: compile vm_opt_and and vm_opt_or. id.def: define id for them to be used in compile.c and vm*.c vm.c: track redefinition of Integer#& and Integer#| vm_core.h: allow detecting redefinition of & and | test/ruby/test_jit.rb: test new insns test/ruby/test_optimization.rb: ditto * Optcarrot benchmark This is a kind of experimental thing but I'm committing this since the performance impact is significant especially on Optcarrot with JIT. $ benchmark-driver benchmark.yml --rbenv 'before::before --disable-gems;before+JIT::before --disable-gems --jit;after::after --disable-gems;after+JIT::after --disable-gems --jit' -v --repeat-count 24 before: ruby 2.6.0dev (2018-09-24 trunk 64821) [x86_64-linux] before+JIT: ruby 2.6.0dev (2018-09-24 trunk 64821) +JIT [x86_64-linux] after: ruby 2.6.0dev (2018-09-24 opt_and 64821) [x86_64-linux] last_commit=opt_or after+JIT: ruby 2.6.0dev (2018-09-24 opt_and 64821) +JIT [x86_64-linux] last_commit=opt_or Calculating ------------------------------------- before before+JIT after after+JIT Optcarrot Lan_Master.nes 51.460 66.315 53.023 71.173 fps Comparison: Optcarrot Lan_Master.nes after+JIT: 71.2 fps before+JIT: 66.3 fps - 1.07x slower after: 53.0 fps - 1.34x slower before: 51.5 fps - 1.38x slower [close https://github.com/ruby/ruby/pull/1963] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64824 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-09-11transform_mjit_header.rb: add static to Init_*k0kubun
as well, to make CI succeed with VM_CHECK_MODE > 1. vm_insnhelper.c: drop unnecessary MJIT_HEADER ifdef. This is intended to be ignored by having `static inline`. Removing that by macro would be helpful for minimizing compilation time, but the impact is not so big and having many MJIT_HEADER check would be bad for maintainability. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64682 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-31transform_mjit_header.rb: more strict FUNC_HEADER_REGEXPk0kubun
to reduce "SKIPPED" logs on MJIT header build. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64144 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-31transform_mjit_header.rb: accept having attributek0kubun
between a type of return value and function identifier. Like: ``` rb_control_frame_t * __attribute__ ((__fastcall__)) rb_vm_opt_struct_aref(rb_execution_context_t *ec, rb_control_frame_t *reg_cfp) { ... } ``` [Bug #14949] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64138 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-24transform_mjit_header.rb: fix performance regressionk0kubun
on r64031. Compiling vm_search_method_slowpath is very slow. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64034 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-05-11transform_mjit_header.rb: workaround for Solaris 10 with old GCCngoto
* tool/transform_mjit_header.rb (MJITHeader.conflicting_types?): Add workaround for Solaris 10 with old GCC (4.6.2), that is essentially the same as for AIX (commit r62326), but probably due to different GCC versions, different error message is shown. [Bug #14751] [ruby-dev:50541] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63400 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-03-17transform_mjit_header.rb: read output and errorsnobu
* tool/transform_mjit_header.rb (MJITHeader.check_code): read output and errors than discarding errors. also cl.exe prints the source file name which is a garbage at this time. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62788 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-03-07transform_mjit_header.rb: inline ivar operationsk0kubun
* Optcarrot before,--jit: ruby 2.6.0dev (2018-03-07 trunk 62689) +JIT [x86_64-linux] after,--jit: ruby 2.6.0dev (2018-03-07 trunk 62689) +JIT [x86_64-linux] last_commit=transform_mjit_header.rb: inline ivar operations Calculating ------------------------------------- before,--jit after,--jit optcarrot 67.852 69.359 fps Comparison: optcarrot after,--jit: 69.4 fps before,--jit: 67.9 fps - 1.02x slower git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62693 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-03-06transform_mjit_header.rb: inline VM instructionsk0kubun
* Optcarrot benchmark (--jit) Before: 62.42 fps After: 65.31 fps git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62677 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-03-06transform_mjit_header.rb: drop obsoleted debug codek0kubun
Now RubyCI is stable for this part. We no longer use this. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62676 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-11common.mk: add comments about r62355 [ci skip]k0kubun
transform_mjit_header.rb: ditto git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62357 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-09transform_mjit_header.rb: fix typo in r62326k0kubun
Today's AIX CI failed due to this... :innocent: git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62332 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-09transform_mjit_header.rb: pass -std=c99 for AIX gcck0kubun
I heard `error: conflicting types for 'restrict'` can be solved by adding `-std=c99`. Ideally we should use the same cflags which are used to compile vm.c, but let me try this and see what happens on AIX. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62326 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-07transform_mjit_header.rb: very verbose error logk0kubun
for debugging AIX..... I have no idea why AIX's gcc is failing to transform MJIT header.... Today's CI output: https://rubyci.org/logs/rubyci.s3.amazonaws.com/aix71_ppc/ruby-trunk/log/20180207T113303Z.log.html.gz#make git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62290 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-07transform_mjit_header.rb: apply Regexp.escapek0kubun
to prevent future breakage Following up r62285 git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62287 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-07transform_mjit_header.rb: ignore unsupported cck0kubun
to generate MJIT header. Even if we can't build MJIT header, Ruby's build should success. And compilers which are not explicitly supported are likely to fail to transform MJIT header. Also you can pass only gcc or clang to --jit-cc=xxx for now. Thus generating header does never make sense. So I decided to conservatively give up MJIT header generation. But please feel free to add your favorite compiler's macro if you think it's working. (Another workaround is passing -D__GNUC__ :p) [Bug #14447] [Bug #14446] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62285 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-06transform_mjit_header.rb: resurrect `exit false`k0kubun
which was just unintentionally dropped at r62252. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62257 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-06transform_mjit_header.rb: add output for debugk0kubun
AIX's header build is failing like: ./miniruby -I./lib -I. -I.ext/common ./tool/transform_mjit_header.rb "/usr/bin/gcc " rb_mjit_header.h .ext/include/powerpc-aix7.1.3.0/rb_mjit_min_header-2.6.0.h error in initial header file: /home/odaira/chkbuild/tmp/build/20180206T113302Z/tmp/20180206-15335556-aaiego.c:19:59: error: conflicting types for 'restrict' extern size_t fread(void *restrict, size_t, size_t, FILE *restrict); ^ https://rubyci.org/logs/rubyci.s3.amazonaws.com/aix71_ppc/ruby-trunk/log/20180206T113302Z.log.html.gz#make It's so hard to know the cause from current output. Let me add debug output and see tomorrow's CI result. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62252 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-06Remove unused capturekazu
Because `Regexp#to_s` add `(?-mix:)`, it is not unnecessary. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62250 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-05transform_mjit_header.rb: restrict transformationk0kubun
to whitelisted patterns. This fix is for NetBSD. [Bug #14439] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62235 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-05transform_mjit_header.rb: same command after errornobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62232 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-05transform_mjit_header.rb: check initial codek0kubun
separating the macros. Applying the kept macros to code which is already affected by the macros may cause errors in initial code. This is hopefully the final fix for icc build failure. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62226 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-05transform_mjit_header.rb: remove predefined macrosk0kubun
semi-automatically. This is basically for icc's __DATE__, __TIME__, and many other families. It causes an error by predefined macro redefinition. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62225 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-05transform_mjit_header.rb: no extra CRsnobu
* tool/transform_mjit_header.rb (MJITHeader.check_code!): open temporary file in binary mode too, not to include extra CRs. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62222 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-05transform_mjit_header.rb: take program name from $0nobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62216 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-05transform_mjit_header.rb: read/write in binmodenobu
* tool/transform_mjit_header.rb: read and write as ASCII-8BIT to make single-byte-optimizable always. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62215 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-05transform_mjit_header.rb: separate macro and codenobu
* tool/transform_mjit_header.rb (separate_macro_and_code): return macro and code separately as the name, and concat before output. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62214 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-05transform_mjit_header.rb: scan by regexpnobu
* tool/transform_mjit_header.rb (find_decl): scan by regexp instead of char-by-char loop. return nil when finished. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62213 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-05Use system option instead of shell redirectkazu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62212 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-04transform_mjit_header.rb: refactor messages and exitnobu
* tool/transform_mjit_header.rb: print non-error messages to STDOUT instead of STDERR. exit with false or abort instead of exit 1. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62205 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-04match whole wordnobu
* tool/transform_mjit_header.rb: add word boundary anchors and match whole word to get rid of false `static` declarations, e.g., rb_str_new_static. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62199 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-04Fix typos and remove redundant `/#{}/`kazu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62198 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-04mjit_compile.c: merge initial JIT compilerk0kubun
which has been developed by Takashi Kokubun <takashikkbn@gmail> as YARV-MJIT. Many of its bugs are fixed by wanabe <s.wanabe@gmail.com>. This JIT compiler is designed to be a safe migration path to introduce JIT compiler to MRI. So this commit does not include any bytecode changes or dynamic instruction modifications, which are done in original MJIT. This commit even strips off some aggressive optimizations from YARV-MJIT, and thus it's slower than YARV-MJIT too. But it's still fairly faster than Ruby 2.5 in some benchmarks (attached below). Note that this JIT compiler passes `make test`, `make test-all`, `make test-spec` without JIT, and even with JIT. Not only it's perfectly safe with JIT disabled because it does not replace VM instructions unlike MJIT, but also with JIT enabled it stably runs Ruby applications including Rails applications. I'm expecting this version as just "initial" JIT compiler. I have many optimization ideas which are skipped for initial merging, and you may easily replace this JIT compiler with a faster one by just replacing mjit_compile.c. `mjit_compile` interface is designed for the purpose. common.mk: update dependencies for mjit_compile.c. internal.h: declare `rb_vm_insn_addr2insn` for MJIT. vm.c: exclude some definitions if `-DMJIT_HEADER` is provided to compiler. This avoids to include some functions which take a long time to compile, e.g. vm_exec_core. Some of the purpose is achieved in transform_mjit_header.rb (see `IGNORED_FUNCTIONS`) but others are manually resolved for now. Load mjit_helper.h for MJIT header. mjit_helper.h: New. This is a file used only by JIT-ed code. I'll refactor `mjit_call_cfunc` later. vm_eval.c: add some #ifdef switches to skip compiling some functions like Init_vm_eval. win32/mkexports.rb: export thread/ec functions, which are used by MJIT. include/ruby/defines.h: add MJIT_FUNC_EXPORTED macro alis to clarify that a function is exported only for MJIT. array.c: export a function used by MJIT. bignum.c: ditto. class.c: ditto. compile.c: ditto. error.c: ditto. gc.c: ditto. hash.c: ditto. iseq.c: ditto. numeric.c: ditto. object.c: ditto. proc.c: ditto. re.c: ditto. st.c: ditto. string.c: ditto. thread.c: ditto. variable.c: ditto. vm_backtrace.c: ditto. vm_insnhelper.c: ditto. vm_method.c: ditto. I would like to improve maintainability of function exports, but I believe this way is acceptable as initial merging if we clarify the new exports are for MJIT (so that we can use them as TODO list to fix) and add unit tests to detect unresolved symbols. I'll add unit tests of JIT compilations in succeeding commits. Author: Takashi Kokubun <takashikkbn@gmail.com> Contributor: wanabe <s.wanabe@gmail.com> Part of [Feature #14235] --- * Known issues * Code generated by gcc is faster than clang. The benchmark may be worse in macOS. Following benchmark result is provided by gcc w/ Linux. * Performance is decreased when Google Chrome is running * JIT can work on MinGW, but it doesn't improve performance at least in short running benchmark. * Currently it doesn't perform well with Rails. We'll try to fix this before release. --- * Benchmark reslts Benchmarked with: Intel 4.0GHz i7-4790K with 16GB memory under x86-64 Ubuntu 8 Cores - 2.0.0-p0: Ruby 2.0.0-p0 - r62186: Ruby trunk (early 2.6.0), before MJIT changes - JIT off: On this commit, but without `--jit` option - JIT on: On this commit, and with `--jit` option ** Optcarrot fps Benchmark: https://github.com/mame/optcarrot | |2.0.0-p0 |r62186 |JIT off |JIT on | |:--------|:--------|:--------|:--------|:--------| |fps |37.32 |51.46 |51.31 |58.88 | |vs 2.0.0 |1.00x |1.38x |1.37x |1.58x | ** MJIT benchmarks Benchmark: https://github.com/benchmark-driver/mjit-benchmarks (Original: https://github.com/vnmakarov/ruby/tree/rtl_mjit_branch/MJIT-benchmarks) | |2.0.0-p0 |r62186 |JIT off |JIT on | |:----------|:--------|:--------|:--------|:--------| |aread |1.00 |1.09 |1.07 |2.19 | |aref |1.00 |1.13 |1.11 |2.22 | |aset |1.00 |1.50 |1.45 |2.64 | |awrite |1.00 |1.17 |1.13 |2.20 | |call |1.00 |1.29 |1.26 |2.02 | |const2 |1.00 |1.10 |1.10 |2.19 | |const |1.00 |1.11 |1.10 |2.19 | |fannk |1.00 |1.04 |1.02 |1.00 | |fib |1.00 |1.32 |1.31 |1.84 | |ivread |1.00 |1.13 |1.12 |2.43 | |ivwrite |1.00 |1.23 |1.21 |2.40 | |mandelbrot |1.00 |1.13 |1.16 |1.28 | |meteor |1.00 |2.97 |2.92 |3.17 | |nbody |1.00 |1.17 |1.15 |1.49 | |nest-ntimes|1.00 |1.22 |1.20 |1.39 | |nest-while |1.00 |1.10 |1.10 |1.37 | |norm |1.00 |1.18 |1.16 |1.24 | |nsvb |1.00 |1.16 |1.16 |1.17 | |red-black |1.00 |1.02 |0.99 |1.12 | |sieve |1.00 |1.30 |1.28 |1.62 | |trees |1.00 |1.14 |1.13 |1.19 | |while |1.00 |1.12 |1.11 |2.41 | ** Discourse's script/bench.rb Benchmark: https://github.com/discourse/discourse/blob/v1.8.7/script/bench.rb NOTE: Rails performance was somehow a little degraded with JIT for now. We should fix this. (At least I know opt_aref is performing badly in JIT and I have an idea to fix it. Please wait for the fix.) *** JIT off Your Results: (note for timings- percentile is first, duration is second in millisecs) categories_admin: 50: 17 75: 18 90: 22 99: 29 home_admin: 50: 21 75: 21 90: 27 99: 40 topic_admin: 50: 17 75: 18 90: 22 99: 32 categories: 50: 35 75: 41 90: 43 99: 77 home: 50: 39 75: 46 90: 49 99: 95 topic: 50: 46 75: 52 90: 56 99: 101 *** JIT on Your Results: (note for timings- percentile is first, duration is second in millisecs) categories_admin: 50: 19 75: 21 90: 25 99: 33 home_admin: 50: 24 75: 26 90: 30 99: 35 topic_admin: 50: 19 75: 20 90: 25 99: 30 categories: 50: 40 75: 44 90: 48 99: 76 home: 50: 42 75: 48 90: 51 99: 89 topic: 50: 49 75: 55 90: 58 99: 99 git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62197 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-04common.mk: install a single header file for JITk0kubun
compilation which is created by transforming a preprocessed vm.c. This file will be used by JIT compiler's generated code which we are going to have from succeeding commits. Makefile.in: generate MJIT header for UNIX environments. win32/Makefile.sub: generate MJIT header for mswin environments. At initial merge, we're going to support only MinGW for Windows. So the header installed by this file won't be used for short term, but we'll add mswin support in a half year or so, for sure. tool/transform_mjit_header.rb: New. This script was originally written as minimize_mjit_header.rb by Vladimir N. Makarov <vmakarov@redhat.com> for Feature 12589. Then I refactored a little so that it can conform CodeClimate CI which is currently set for Ruby's GitHub repository, and fixed some bugs and ported it to work on Windows. Also, as original minimize_mjit_header.rb takes too long time to run, this is modified to skip minimization step because having *static* unused definitions does not waste compilation time on -O2 since compiler can skip to compile unused static functions. So this does no longer "minimize" the header and is renamed. This header installation does NOT include a header to automatically export symbols used by MJIT. That's because original MJIT code was failing to export symbols in the import header in macOS environment. But I would like to have the functionality for maintainability in the future. I'll manually export things but it would be just an intemediate solution. Patch by: Vladimir N. Makarov <vmakarov@redhat.com> Part of: Feature 12589 and 14235. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62187 b2dd03c8-39d4-4d8f-98ff-823fe69b080e