Hastega (will be renamed) concludes NOT to adopt to be implemented using High Performance Erlang (HiPE)

Thank ElixirConf for giving me another chance to make a presentation at ElixirConf US 2019:

This presentation will be conducted by me and Mr. Hisae, who is a graduate student in my laboratory and a co-author of Hastega. He's a great meta-programmer because he wrote a new feature of Hastega only one night! Look forward to an announcement on it at ElixirConf US 2019!

BTW, development of Hastega has been smooth but a little late. Unfortunately and unexpectedly, we can’t release the 1st practical use version of it until June 1st, 2019, when we’ve made a presentation on Erlang & Elixir Fest 2019.

We've made a smooth progress as follows:

1. Analyze abstract syntax trees (ASTs), identify code patterns in the Elixir Zen styled code, and optimize them in the syntax level, if necessary;
2. Establish native code with SIMD instructions that can be executed efficiently;
3. Establish basis technologies on code generation

However, it is a bottle-neck of the progress that we decide how to generate and execute the native code (2) based on the analysis, identification and optimization (1). It should be decided carefully because it may be a quite fatal performance bottle-neck of Hastega.

We’ve considered such various approaches. We’ve just found and concluded one of our approaches was able to realize but too inefficient to be adopted for Hastega, at shinjuku.ex on May 31, 2019. The unadopted approach uses High Performance Erlang (HiPE).

We're currently investigating other approaches, and can decide to adopt one of them, soon. Then, I think we will be able to develop Hastega smoothly, again.

Thus, I'll show you the knowledge on HiPE, which I investigated.

Firstly, I saw the paper of HiPE: http://erlang.org/workshop/2003/paper/p36-sagonas.pdf.

Fig. 1 in the paper shows the structure of a HiPE-enabled Erlang/OTP system.

A good idea flashed into my mind! How about modifying elixirc to generate native code at compile time, attach it to BEAM files, and make it run on HiPE-enabled Erlang?

I thought that the advantage of the HiPE approach may be that it would be realized only with modifying elixirc. It wouldn’t require to modify Erlang VM if it is HiPE-enabled.

I also guessed that it might have more performance potential than NIFs because it might be able to be executed unlimitedly, unlike NIFs.

I told this idea to José by e-mail, soon.

However, he informed me the original HiPE announcement: http://erlang.2086793.n4.nabble.com/HiPE-in-OTP-22-td4725613.html#a4725614

This told me that HiPE won't be supported. I was shocked by this statement!

I thought that if "the HiPE hook", which is a calling point from the Erlang interpreter to HiPE native code, will be kept even if HiPE will be disabled, I can implement Hastega by the HiPE approach. If it is feasible, I was going to ask the OTP team to keep the HiPE hook. Thus, I had to make sure of the feasibility of the HiPE approach.

Thus, I investigated the source code of HiPE at the shinjuku.ex on May 31th, 2019: https://github.com/erlang/otp/tree/master/lib/hipe

My investigation at that time focused on the HiPE hook, and I succeeded in identifying it, at last! I'll show the chain of functions that it calls:

1. The HiPE hook is "HIPE_MODE_SWITCH" in the opcodes in otp/erts/emulator/hipe/hipe_instrs.tab;
2. The function hipe_mode_switch in otp/erts/emulator/hipe/hipe_mode_switch.c;
3. The inlined function hipe_call_to_native in otp/erts/emulator/hipe/hipe_x86_glue.h (in the case of x86 or amd64);
4. The assembly coded function x86_call_to_native in otp/erts/emulator/hipe/hipe_amd64_glue.S (in the case of amd64);
5. The stub generated by PN_CALLEE in otp/erts/emulator/hipe/hipe_mkliterals.c into struct hipe_process_state hipe in otp/erts/emulator/beam/erl_process.h;
6. The member ncallee of struct hipe_process_state in otp/erts/emulator/hipe/hipe_process.h

The ncallee is set by native_address in otp/erts/emulator/beam/beam_load.c, which is called by get_module_info in otp/erts/emulator/beam/beam_load.c.

I found that the chain of the functions called by the HiPE hook is too long and complicated to achieve enough efficiency for Hastega.

I also found that both HiPE code and NIFs may be invoked within a new independent process generated by the caller process. If they run longer than 1ms and aren’t under the dirty scheduler, they may be scheduled very occasionally. I guessed that this may be the 1ms constraint mechanism. This means that not only NIFs but also HiPE may be under the 1ms constraint mechanism.

I can't accept the complication and the constraint because Hastega pursues more and more performance!!! Thus, I abandon the plan to adopt HiPE for Hastega.

Don’t worry, however!! We’re currently investigating other approaches. I’m sure that these approaches are more hopeful!!!

Finally, we'll rename Hastega. We're discussing on a new name of it (sorry, in Japanese):

In fact, we'll rename Hastega. It's NOT because SQUARE ENIX pressers us to do so. Because we should respect on it, we thought we should follow the regular procedure. However, we'll rename Hastega because we don't wanna continue to use the name even with paying the effort.

The new name of Hastega will be shown in our presentation on ElixirConf US 2019, and we hope to make it sure that the 1st practical use version of the renamed Hastega can be released until it, thank you. I'm looking forward to see you at ElixirConf US 2019!