[cwn] Attn: Development Editor, Latest OCaml Weekly News
Alan Schmitt
alan.schmitt at polytechnique.org
Tue Nov 15 04:28:26 PST 2022
Hello
Here is the latest OCaml Weekly News, for the week of November 08 to 15,
2022.
Table of Contents
─────────────────
OCaml Platform Installer alpha release
Building the OCaml Toolchain with Bazel - PoC
OCaml compiler development newsletter, issue 6: March 2022 to September 2022
diskuvbox: small set of cross-platform CLI tools
http_async 0.2.0
Other OCaml News
Old CWN
OCaml Platform Installer alpha release
══════════════════════════════════════
Archive:
<https://discuss.ocaml.org/t/ann-ocaml-platform-installer-alpha-release/10652/10>
Continuing this thread, Paul-Elliot explained
─────────────────────────────────────────────
Let me try to explain a little bit more!
So, there are two distinct process:
1. Installing `ocaml-platform'. This is done by the installer script
which, as you found out, also installs opam.
2. Installing the platform tools: `merlin', `dune', `ocamlformat', …
This is done by running `ocaml-platform' (more explanation of what
happens during a run of `ocaml-platform' below).
For 1., I agree that using a script is not ideal at all. We do want to
have a system package, ideally as part of a distribution. However, we
are really at an early stage, and first focused on developing the
“core” functionality, for feedback before going further. Note that
`sudo' is only used when needed (for the `install' command), and that
we encourage users to check what is inside the script: Taken from the
README:
Don’t hesitate to have a look at what the script does. In
a nutshell, the script will install a static binary into
`/usr/local/bin/ocaml-platform' as well as the `opam'
binary if it isn’t already installed.
For 2. here is what happens when you run `ocaml-platform' (also
detailed in the [README]):
• First, “driving” `opam', `ocaml-platform' infer a version of each
platform tools (`dune' `merlin' etc). It will use the best one
available for your version of OCaml, except for `ocamlformat' where
it will first search for a `.ocamlformat' with a specified version.
• Then, `ocaml-platform' checks which of the `tools.version' it has in
a local cache (which is initially empty).
• Using `opam', it builds in a “sandbox switch” the ones to build (the
ones that are not already in the cache)
• it populates the local cache with the newly built package
• Using `opam', it installs everything from the local cache (which is
a local opam repo).
So, to answer your question:
What happens when a new version of dune is released, how
do I get an update using “ocaml-platform-installer”?
You simply need to run `ocaml-plaform', it will know that there is a
new version of `dune' by asking `opam' the best version available. (I
answered incorrectly the first time as I thought you were specifically
speaking about a new version of `ocaml-platform', my apologies!)
> In fact, currently only the `ocaml-platform' binary is
being downloaded.
When I look into `installer.sh', at least also `opam' is
downloaded and “installed” (copied to PREFIX, there’s a
lack of manpages etc.).
Yes, I forgot about opam’s installation… I meant that only
`ocaml-platform' (and `opam'…) is downloaded _through the install
script_. Other tools are downloaded through opam.
> We do have security in mind!
Could you elaborate on that?
I simply meant that, when discussing how the tool would work, we tried
not to miss security issues. Since most of the communication with the
outside world is done through opam, we felt it was okay, but I am no
security expert… and would love specific advices on the matter :)
@hannes I hope that I answered your questions in a clearer way!
It looks like the `ocaml-platform' tool would be
beneficial even if you already have `opam' and a
development environment, as it seems to offer a convenient
way to share various developer tools between multiple
`opam' switches (and it could perhaps build and install
itself as one of such tools, so it can be conveniently be
kept up to date).
@edwin that’s a good point! There are still some things to work out:
since the tool is not switch dependent, having multiple versions of it
(even in multiple switches) might create problem (for instance, v2
updates the cache to the v2 version, and v1 cannot use the cache
anymore. (Similar to opam 1 and 2 with the update of `.opam'!))
[README] <https://github.com/tarides/ocaml-platform-installer#readme>
Building the OCaml Toolchain with Bazel - PoC
═════════════════════════════════════════════
Archive:
<https://discuss.ocaml.org/t/ann-building-the-ocaml-toolchain-with-bazel-poc/10776/1>
Gregg Reynolds announced
────────────────────────
I’m inordinately pleased to announce that a Proof-of-Concept Bazel
build of the OCaml compilers and tools (latest trunk version) is
available for testing and exploration. Tested on MacOS 12.6 and Ubuntu
20.04.5 LTS. It is available at
<https://github.com/obazl-repository/ocaml>
This uses a stripped-down version of [rules_ocaml], embedded in the
repo in subdirectory `bzl'.
Currently the following targets build and run: ocamlc, ocamlopt.byte,
ocamllex, all of the tools in the `tools/' subdirectory.
Supports Zig as an alternate C toolchain and as a cross-compiler. I
was also able to use the LLVM toolchain at
<https://github.com/grailbio/bazel-toolchain>, but that was months ago
and I have not tested it with this new version. But the code is in the
WORKSPACE.bazel file, so getting it to work would be a good starter
project.
Cross-compilation (using Zig) is supported for the C code. Full
support is going to take some more work. For example, to compile a
linux runtime on a mac:
`$ bazel build boot/bin:ocamlrun --config=mac_linuxamd64'
It uses some interesting Bazel features. For example, it uses
platforms and toolchains to support the various compilers - instead of
building ocamlc.byte or ocamlopt.byte, you build one target
(boot/compiler) and pass CLI args telling Bazel which platform (vm or
sys) should host the build compiler and which should be the target.
Similarly, debug and instrumented variants are controlled by
parameters rather than separate targets. It also supports fine-grained
control of compile/link flags, so developers should be able to
optimize for diffferent scenarios. You can tell it whether or not to
compile .mli files separately. You can pass a custom `primitives' file
for use with `-use-prims' and it will be used everywhere. Etc.
It’s far from polished but I think it works well enough for people to
do a little testing and exploration.
Even if you have no interest in Bazel you may find the notes in the
bzl/docs subdir worth looking at. I spent a /lot/ of time studying the
Makefiles and trying to understand the build, and took pretty
extensive notes. They’re not very well organized but they have a lot
of info, I think.
Feedback welcome. I’m not sure how far I’ll go with this, but I at
least want to get complete cross-compilation working, and I’d also
really like to get persistent workers going.
[rules_ocaml] <https://github.com/obazl/rules_ocaml>
Danielo Rodríguez asked and Gregg Reynolds replied
──────────────────────────────────────────────────
Is this bazel configuration intended to be used in ocaml
projects or it is a way to cross-compile the ocaml tools
itself?
Only for building the compilers. The general rules_ocaml ruleset must
support a lot of stuff that is not needed for building the compilers
(e.g. ppx support), and the build protocol for bootstrapping compilers
is a little more complicated than the standard build protocol. For
that very reason (among others), compiler developers need a build
program that they can understand and modify with reasonable effort. So
I decided to make the OCaml boot rules as minimal and understandable
as possible (I forgot to add “maintinability” in the Goals section of
the readme.) I think eventually the Bazel rules (including the
implementation code) will be pretty easy to understand; I’m currently
cleaning up and refactoring the source code to get rid of all the
cruft left over from development.
I think there is a very felicitous side-effect of all this, which is
that the Bazel build program makes the build structure very clear and
pretty simple, and Bazel features (like querying) make it very easy to
explore and experiment. So much so that I can envision that it could
be used in introductory OCaml material. Why not use the source code of
the compiler itself to learn not only how to program in OCaml but how
to organize and manage code?
To /use/ a compiler built using these rules one could write some
installation rules, or in a Bazel-based project use a toolchain that
depends on it (rather than an OPAM-installed compiler, which is what
rules_ocaml currently uses.)
OCaml compiler development newsletter, issue 6: March 2022 to September 2022
════════════════════════════════════════════════════════════════════════════
Archive:
<https://discuss.ocaml.org/t/ocaml-compiler-development-newsletter-issue-6-march-2022-to-september-2022/10777/1>
gasche announced
────────────────
I’m happy to publish the sixth issue of the “OCaml compiler
development newsletter”. You can find all issues using the tag
[compiler-newsletter ].
Note: the content of the newsletter is by no means exhaustive, only a
few of the compiler maintainers and contributor had the time to write
something, which is perfectly fine.
Feel free of course to comment or ask questions!
If you have been working on the OCaml compiler and want to say
something, please feel free to post in this thread! If you would like
me to get in touch next time I prepare a newsletter issue (some random
point in the future), please let me know by Discuss message or by
email at (gabriel.scherer at gmail).
[compiler-newsletter ]
<https://discuss.ocaml.org/tag/compiler-newsletter>
Context
╌╌╌╌╌╌╌
The Multicore merge is behind us now. We are in the final preparation
stages for 5.0 (but by no means the end of the Multicore-related work,
many things were left to do in 5.1 and further releases). The
non-Multicore-development has been restarting slowly but surely.
@yallop Jeremy Yallop
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
We’re starting up the [modular macros] work at Cambridge again, with
the aim of adding support for typed, hygienic, compile-time
computation to OCaml. Back in 2015 we presented our original design at
the OCaml Users and Developers Workshop, and we subsequently went on
to develop a prototype in a branch of the OCaml compiler. We’re
planning to complete, formalise, and fully implement the design in the
coming months.
[modular macros]
<https://www.cl.cam.ac.uk/~jdy22/projects/modular-macros/>
@dra27 David Allsopp
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
Various bits of house-keeping on the compiler distribution have been
managed for 5.0, taking advantage of the major version increment. All
the compiler’s C symbols are now prefixed `caml_', vastly reducing the
risks of conflicts with other libraries (#[10926] and #[11336]). The
5.x compiler now installs all of its additional libraries (Unix, Str,
etc.) to separate directories, with a friendly warning that you need
to specify `-I +unix' etc. which makes life slightly easier for build
systems (#[11198]) and the compiler now also ships `META' files for
all its libraries by default (#[11007] and #[11399]). Various other
bits of 5.0-related poking include a deprecation to allow the
possibility in future for sub-commands to the `ocaml' command. Instead
of `ocaml script', one should now write `ocaml ./script' (#[11253]).
The compiler’s bootstrap process (this is the mechanism which updates
the initial compiler in `boot/ocamlc' which is used to build the
compiler “from cold”) is now reproducible (#[11149]), easing the
review process for pull requests which need to include changes to the
boot compiler. Previously we required a core developer to re-do the
bootstrap and then separately merge the work, where now a core
developer can merely pull the branch and check that the committed
artefact is reproducible.
Looking beyond the release of OCaml 5.0, I’ve also been working to
resurrect the disabled Cygwin port of OCaml (#[11642]) and, more
importantly, getting the MSVC native Windows port working again
(that’s still WIP!).
At the OCaml Workshop this year, I demonstrated my “relocatable
compiler” project, which aims both to eliminate various kinds of
“papercut” when using bytecode executables but, much more importantly,
allows compiler installations to be copied to new locations and still
work, which paves the way for faster creation of opam switches
_everywhere_. It was great to be able to meet so many people in person
in Slovenia for the first in-person workshop since 2019, but
unfortunately that came at the cost of catching COVID, which has
slowed me down for the weeks since! The next stage for the relocatable
compiler is to have an opam remote which can be added to allow opt-in
testing of it with OCaml 4.08-5.0 and then to start opening pull
requests hopefully for inclusion of the required changes in OCaml 5.1
or 5.2.
[10926] <https://github.com/ocaml/ocaml/pull/10926>
[11336] <https://github.com/ocaml/ocaml/pull/11336>
[11198] <https://github.com/ocaml/ocaml/pull/11198>
[11007] <https://github.com/ocaml/ocaml/pull/11007>
[11399] <https://github.com/ocaml/ocaml/pull/11399>
[11253] <https://github.com/ocaml/ocaml/pull/11253>
[11149] <https://github.com/ocaml/ocaml/pull/11149>
[11642] <https://github.com/ocaml/ocaml/pull/11642>
@sadiqj Sadiq Jaffer
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
The bulk of my upstream work over the last year in OCaml 5.0 has been
on Runtime Events, a new tracing and metrics system that sits inside
the OCaml runtime. The initial PR can be found at #[10964] and there
was a separate documentation PR in #[11349]. Lucas Pluvinage has
followed up with PR #[11474] which adds custom application events to
Runtime Events and I hope isn’t too far off merging. We gave a talk at
the OCaml Users and Developers workshop on Runtime Events and I’m
hoping there will be a video up on that soon.
[10964] <https://github.com/ocaml/ocaml/issues/10964>
[11349] <https://github.com/ocaml/ocaml/issues/11349>
[11474] <https://github.com/ocaml/ocaml/issues/11474>
@garrigue Jacques Garrigue
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
We have continued our work on refactoring the type checker for clarity
and abstraction. An interesting result was PR #[11027]: separate
typing of counter-examples from `Typecore.type_pat'. Namely, around
2015 `type_pat' was converted to CPS style to allow a more refined way
to check the exhaustiveness of GADT pattern-matching. A few more
changes made the code more and more complex, but last year in #[10311]
we could extract a lot of code as case-specific constraints. This in
turn made possible separating `type_pat' into two functions:
`type_pat' itself, only used to type patterns in the source program,
which doesn’t need backtracking, and `check_counter_example_pat', a
much simpler function which works on counter examples generated by the
exhaustiveness checker. I have also added a `-safer-matching' flag for
people who don’t want the correctness of compilation to depend on the
subtle typing arguments involved in this analysis (#[10834]). In
another direction, we have reorganized the representation of type
parameter variances, to make the lattice involved more explicit
(#[11018]). We have a few PRs waiting for merging: #[11536] introduces
some wrapper functions for level management in types, #[11569] removes
the encoding used to represent the path of hash-types associated with
a class, as it was not used in any meaningful way.
There are also a large number of bug fixes (#[10738], #[10823],
#[10959], #[11109], #[11340], #[11648]). The most interesting of them
is #[11648], which extends `type_desc' to allow keeping expansions
inside types. This is needed to fix a number of bugs, including
#[9314], but the change is invasive, and reviewing may take a while.
Coqgen, the Coq backend (previously named ocaml_in_coq), is still
progressing, with the addition of constructs such as loops, lazy
values and exceptions [Coqgen], and we are trying to include GADTs in
a comprehensive way.
[11027] <https://github.com/ocaml/ocaml/issues/11027>
[10311] <https://github.com/ocaml/ocaml/issues/10311>
[10834] <https://github.com/ocaml/ocaml/issues/10834>
[11018] <https://github.com/ocaml/ocaml/issues/11018>
[11536] <https://github.com/ocaml/ocaml/issues/11536>
[11569] <https://github.com/ocaml/ocaml/issues/11569>
[10738] <https://github.com/ocaml/ocaml/issues/10738>
[10823] <https://github.com/ocaml/ocaml/issues/10823>
[10959] <https://github.com/ocaml/ocaml/issues/10959>
[11109] <https://github.com/ocaml/ocaml/issues/11109>
[11340] <https://github.com/ocaml/ocaml/issues/11340>
[11648] <https://github.com/ocaml/ocaml/issues/11648>
[9314] <https://github.com/ocaml/ocaml/issues/9314>
[Coqgen] <https://github.com/COCTI/ocaml/pull/3>
@gasche Gabriel Scherer
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
@nojb Nicolás Ojeda Bär has done very nice work on using
tail-modulo-cons for some List functions of the standard library,
which I helped review along with Xavier Leroy:
• #[11362]: List.map, List.mapi, List.map2
• #[11402]: List.init, List.filter, List.filteri, List.filter_map
Some of those functions were hand-optimized to use non-tail-rec code
on small inputs. Nicolás’ micro-benchmarks showed that often the
TMC-transformed version was a bit slower on very small lists, up to
20% slower on lists of less than five elements. We wanted to preserve
the performance of the existing code exactly, so we did some manual
unrollling in places. (The code is a bit less readable than the
obvious versions, but much more readable than was there before.)
I worked on fixing a 5.0 performance regression for bytecode-compiled
programs (#[11337] ). I started with the intuition that the overhead
came from having a concurrent skip list in the 5.x runtime instead of
a non-concurrent skip list in the 4.x runtime, and wrote tricky code
to use a sequential skip list again. Soon I found out that the
performance regression was due to something completely different and
had to dive into the minor-root-scanning code.
When I started looking at the multicore runtime, I had no idea how to
print a backtrace from a C segfault without using `valgrind'. I wrote
some documentation on debugging in [runtime/HACKING.adoc] in the hope
of helping other people.
I spent some time reading `lambda/switch.ml', which compiles
shallow-match-on-constructor-tags into conditionals and jump tables.
The file contains some references to research papers from the 90s, but
it was unclear to me how they connected to the implementation. After a
nice discussion with Luc Maranget I could propose a documentation PR
#[11446] to explain this in the source code itself. Thanks to Vincent
Laviron for the nice review – as always.
[11362] <https://github.com/ocaml/ocaml/issues/11362>
[11402] <https://github.com/ocaml/ocaml/issues/11402>
[11337] <https://github.com/ocaml/ocaml/issues/11337>
[runtime/HACKING.adoc]
<https://github.com/ocaml/ocaml/blob/8796a4f4f0e4450795ee0bae2fb5bb32d313e5c7/runtime/HACKING.adoc>
[11446] <https://github.com/ocaml/ocaml/issues/11446>
@gadmm Guillaume Munch-Maccagnoni
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
(written by @gasche)
Guillaume worked on updating the “GC timing hooks” and
`caml_scan_roots_hook' of the OCaml runtime to be multicore-safe, and
added a new hook `caml_domain_terminated_hook'. (#[10965], #[11209])
We rely on runtime hooks in our experimental [boxroot] library, and
updating hooks for 5.0 was necessary to have a correct 5.0 version of
boxroots.
Also related to our boxroot experiments, Guillaume wanted an efficient
way to check whether a domain thread was currently holding its runtime
lock – it does not when executing long non-OCaml computations that do
not access the OCaml runtime. Guillaume changed the `Caml_state' macro
to provoke an error when accessing the domain state without holding
the domain runtime lock – a programming mistake that could easily go
unnoticed before in simple testing and crash in hard-to-debug ways on
different inputs – and introduced a new `Caml_state_opt' macro that is
NULL when the runtime lock is not held. (#[11138], #[11272],
#[11506]).
Guillaume worked on quality treatment of asynchronous actions in the
new Multicore runtime. (#[10915], #[11039], #[11057], #[11095],
#[11190]). Asynchronous actions are callbacks called by an OCaml
program by the environment (instead of an explicit request by the
programmer at the point they happen). They include for example signal
handlers, finalizers called by the GC, Statmemprof callbacks.
Supporting them well requires tricky code, because the runtime must
ensure that such actions are executed promptly, but in a context where
running OCaml code is safe. (For example it is easy to have bugs where
a asynchronous action raises an exception in the middle of a runtime
function that is not exception-safe.) The 4.x runtime had a lot of
asynchronous-action fixes between 4.10 and 4.14, but sadly many of
these improvements were not backported in the Multicore branch (they
required expert adaptation to a very different runtime codebase), and
were thus lost in the Multicore merge. The present work tries to come
back to a good state for 5.0 and 5.1 – some of the fixes were
unfortunately not merged in time for 5.0. Statmemprof support is
currently disabled for 5.x, and this work will also be useful for
Statmemprof.
[10965] <https://github.com/ocaml/ocaml/issues/10965>
[11209] <https://github.com/ocaml/ocaml/issues/11209>
[boxroot] <https://gitlab.com/ocaml-rust/ocaml-boxroot/>
[11138] <https://github.com/ocaml/ocaml/issues/11138>
[11272] <https://github.com/ocaml/ocaml/issues/11272>
[11506] <https://github.com/ocaml/ocaml/issues/11506>
[10915] <https://github.com/ocaml/ocaml/issues/10915>
[11039] <https://github.com/ocaml/ocaml/issues/11039>
[11057] <https://github.com/ocaml/ocaml/issues/11057>
[11095] <https://github.com/ocaml/ocaml/issues/11095>
[11190] <https://github.com/ocaml/ocaml/issues/11190>
Guillaume Munch-Maccagnoni added
────────────────────────────────
Thanks for the summary @gasche. (@gasche asked me to write about what
I did; it took me too long to do it, so he wrote something for me.) In
addition to what @gasche described, many things I have worked on are
not visible in the change log, notably bug fixing and cleanups for the
new multicore runtime.
Systhreads
╌╌╌╌╌╌╌╌╌╌
The runtime lock check feature was long-time requested by some
foreign-function interface (FFI) users, also its implementation sent
me into a rabbit hole of fixes with the new systhread implementation
(#[11271], #[11385], #[11386], #[11390], #[11473], #[11403]), with
some fixes still pending. The multicore systhread implementation was
new code with few experts, so this made it benefit from needed
attention.
[11271] <https://github.com/ocaml/ocaml/issues/11271>
[11385] <https://github.com/ocaml/ocaml/issues/11385>
[11386] <https://github.com/ocaml/ocaml/issues/11386>
[11390] <https://github.com/ocaml/ocaml/issues/11390>
[11473] <https://github.com/ocaml/ocaml/issues/11473>
[11403] <https://github.com/ocaml/ocaml/issues/11403>
Memory model
╌╌╌╌╌╌╌╌╌╌╌╌
In a discussion in a conference call with OCaml developers shortly
after the merge (to which I was invited thanks to @kayceesrk), I was
asking whether the accesses to the OCaml heap from C (runtime, VM and
FFI) should not be made atomic. At the time, the C parts of the
runtime were accessing the OCaml heap through non-atomic loads and
stores. (More generally, adapting the OCaml runtime and existing FFI
code to multicore requires explaining first how to implement the OCaml
memory model in the C memory model.) I do not remember the exact
wording (and I would not be able to quote it given that the meeting
was unfortunately non-public), but I remember that the answer did not
convince me.
I later came back with concrete examples of undefined behaviour
involving the `Field' macro (#[10992]). This put it back on the radar
and enabled OCaml developers to start addressing this issue.
Unfortunately, it appears that the question of the OCaml memory model
for C code will still not be clarified as of 5.0.
The challenge is finding a good balance between
backwards-compatibility, efficiency and standards-compliance. A first
part was addressed by marking accesses via the `Field' macro
`volatile' (#[11255]). The C `volatile' keyword does not imply
atomicity according to the standard, but it is used in many legacy
codebases in this way, so this is likely to work while remaining
(mostly) backwards-compatible. (To my knowledge only `Field' which I
gave as an example has been fixed. I suggested that all runtime, VM
and FFI API should be reviewed in the new light, but I am not aware
that developers had the time to do this yet. Similarly, user C code
that is not using strictly the documented FFI and instead relies on
knowledge of the OCaml value representation in order to access values,
will have to be audited before any use inside multicore programs.)
Another issue still to be addressed is synchronizing the read of
initial writes to values, as needed for memory safety (this problem
does not appear in the PLDI2018 paper, because it does not accurately
model initial writes). Even leaving backwards-compatibility for the
FFI aside, using `memory_order_acquire' on mutable loads would be
cheap on x86-64 but expensive on Arm. Instead, OCaml relies on some
other order-preserving property that comes for free on Arm (address
dependency). Now, C compilers do not understand this notion yet
([McKenney 2017]), let alone let OCaml offer a backwards-compatible
and standards-compliant solution.
• [One solution] which @kayceesrk and I proposed involves requiring
users to change reads of mutable fields to using a different
dedicated function or macro (doing an acquire atomic load), with
some provisions that they can adapt their program to multicore
gradually (e.g. legacy code still works due to absence of
races/parallelism). (This is a milder situation in terms of
backwards-compatibility than the one with the original concurrent
multicore GC that led to its abandonment in favour of the
stop-the-world design.)
• Another path I proposed was to rely on _de facto_ preservation of
dependency ordering by current compilers (modulo a certain number of
constraints and fingers-crossings), taking inspiration from the
“Linux Kernel memory model” (which as it names does not indicate,
denotes the absence of one); and to subsequently work towards
specifying what this means in terms of OCaml’s requirements. This
would be the ideal approach, because it does not require legacy
programs to be fixed. I had the chance to discuss this approach with
@stedolan in Ljubljana in September, and without entering into
technical details, one conclusion was that it was possibly
interesting work, but very hypothetical. Moreover, other users of
dependency ordering (e.g. Linux RCU) have different requirements,
and are currently pushing propositions to evolve C/C++ that are
going in directions that are not necessarily suited to OCaml’s needs
on this issue.
The risk is that OCaml programs end up in a no-man’s-land in terms of
C standards due to the `volatile' fix working well enough (so far; as
far as we know; etc.), and this even in the event where a purely
standards-compliant solution was later adopted by OCaml (but where
users would lack motivation to adapt their code).
This issue has not received as much discussion as I had hoped; there
was no feedback on the solutions. Unfortunately, this problem not
being addressed for 5.0 means that OCaml developers might be
envisioning breaking FFI changes later on, including changes that
could cause API breakages in the multicore OCaml-Rust interface which
I have been working on.
[10992] <https://github.com/ocaml/ocaml/issues/10992>
[11255] <https://github.com/ocaml/ocaml/pull/11255>
[McKenney 2017] <https://arxiv.org/abs/1701.00854v5>
[One solution]
<https://github.com/ocaml/ocaml/issues/10992#issuecomment-1112394309>
Lazy in multicore
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
I came up with a design for thread-safe Lazys in multicore that would
allow various efficient (and custom) synchronization schemes
([ocaml-multicore#750]) and I started to implement a prototype. Lazy
is currently thread-unsafe in multicore (although memory-safe);
thread-safe lazys would allow to implement optimally globals with
delayed initialization, one common and convenient form of
synchronization in multithreaded languages (cf. “magic statics” in
C++11, lazy statics in Rust), along with more forms of lazy that are
useful in theory but that have not been tried in practice yet.
Unfortunately there is a tedious [bootstrapping issue] with my
prototype, and I have not made progress since (help is welcome).
Another lack of motivation comes from the fact that the lazy
short-cutting has been removed in OCaml 4.14 for fear it was too
expensive for GC marking. If not for short-cutting, it would be
possible to implement lazy in a much simpler way directly in OCaml. So
the runtime is currently in a weird state where 95% of the invasive
work to have the short-cutting optimization is still there, but the
optimization is not done (actually, only happens for lazys that are
forced while still young). Luckily, a side-effect of my work presented
at the OCaml workshop is to show that lazy short-cutting comes
essentially for free during GC marking thanks to instruction-level
parallelism on modern processors, so I expect the core OCaml
developers to put it back in.
[ocaml-multicore#750]
<https://github.com/ocaml-multicore/ocaml-multicore/issues/750#issuecomment-978125441>
[bootstrapping issue]
<https://github.com/ocaml/ocaml/issues/11182#issuecomment-1098275032>
Pooled mode
╌╌╌╌╌╌╌╌╌╌╌
The work on Boxroot was also the occasion to see whether the obscure
“pooled” mode whereby OCaml frees memory for you (`OCAMLRUNPARAM=c'),
was useful to us. I had a PR to fix bugs in this mode and to report
other design issues, but nobody was reviewing it after a year so I
closed the PR. My reasoning is that since the mode is unmaintained and
its design is broken, we can as well advise FFI users to steer clear
from this mode.
Other projects
╌╌╌╌╌╌╌╌╌╌╌╌╌╌
@gasche’s original message mentioned my work on restoring some safety
guarantees of asynchronous callbacks in OCaml multicore. It is in fact
a spin-off of past work where I aimed to show that it is possible to
mix certain systems programming requirements (correctly releasing
resources and handling errors) with certain functional programming
requirements (being able to interrupt computation asynchronously),
given the right language design. (My work on memprof-limits was
another part of this work.) AFAIU OCaml multicore was originally
developed with the idea that it is not really possible to handle
asynchronous interruptions nicely at a language-design level, which
explains that they were given second-class status in the new runtime,
despite many existing programs using them.
The work on Boxroot presented with @gasche at the ML workshop is meant
to enable the development of safe interfaces between OCaml and Rust. I
am working on such an interface for multicore OCaml to serve as a
common building block for the few other projects in this area, by
giving a reference implementation of the multicore memory-safety
model. We also opened an RFC opening the possibility to integrate
Boxroot with the OCaml runtime, but there are a few things to clarify
before this can be considered.
At the OCaml workshop I presented an implementation in the runtime of
a large page allocator with support for huge pages (THP) and efficient
pointer classification (i.e. what you need to point to
dynamically-allocated memory in the GC heap with sharing-like
properties, à la Ancient and OCamlnet). It is part of longer-term
goals, along with Boxroot, aiming to show how it is possible to mix
tracing GC with the idea of _linear allocation with re-use_ coming
from linear logic (a.k.a. “functional but in place” in recent papers).
(Such a hybrid allocation scheme could open now possibilities in
programming for large data or low latency requirements in functional
programming.) This is loosely inspired by old papers by Jean-Yves
Girard on mixing linear, intuitionistic and classical reasoning inside
the same deductive system.
All this is part of a longer-term investigation into the mixing of
systems and functional programming centred on the notion of resource
as a first-class value. (I have various ideas in this area ranging
from hands-on to very theoretical; students in France or in Cambridge
UK should feel free to get in touch with me if they are interested
overall by this subject.)
gasche replied
──────────────
Luckily, a side-effect of my work presented at the OCaml
workshop is to show that lazy short-cutting comes
essentially for free during GC marking thanks to
instruction-level parallelism on modern processors, so I
expect the core OCaml developers to put it back in.
My understanding of this issue is as follows:
• None of the core developers is actively working on multicore lazy,
so you would probably have to contribute yourself if you want to
improve them. (I have on the occasion tried to help you along, and
would be happy to do it again, but right now I would rather focus on
boxroot.)
• The reason why shortcutting was disabled is the interaction with GC
prefetching which, as you explain, is currently missing from
Multicore. Prefetching is important for performance in 4.x, and Jane
Street would like to see it back in 5.x as well. Someone at Tarides
is working on this (I don’t know who, KC would). So it sounds likely
that prefetching will make a comeback for 5.x, and one would have to
adapt the implementation again to work with lazy shortcutting.
You posted your patch to the prefetching code to re-enable lazy
shortcutting in
<https://github.com/ocaml-multicore/ocaml-multicore/issues/750#issuecomment-986847502>,
with preliminary benchmarks suggesting that it does not decrease the
performance of marking. On the same issue, @lpw25 wondered whether we
really need shortcutting (what lazy-using programs actually depend on
it for performance), suggesting that it is not a high-priority issue.
One difficulty that I foresee is that there is no one strongly pushing
for lazy shortcutting to come back (this somewhat supports Leo’s
point), and at the same time it is hard to convince the Jane Street
people that the prefetching change you suggest is innocuous for their
workflows (because only they can run the benchmarks they care about,
and apparently they are not motivated enough by this issue to run them
with your patch).
Richard Eisenberg said
──────────────────────
The Jane Street compilers team is hard at work on a number of fronts,
though perhaps less visibly than in the past. Previously, we worked on
a feature and got it merged upstream without releasing it internally.
This meant that our colleagues would have to wait for a main trunk
release of OCaml before getting the new feature, a turnaround time
that was sometimes too slow. In order to decrease latency between
feature conception and use, we have now changed to a new workflow
where we develop and release features internally while we work to
upstream.
Not only does this new workflow get features into the hands of our
developers faster, it also allows us to improve the final product:
• The upstreaming discussion can be informed by actual practice by the
700+ OCaml developers within Jane Street. We can, for example,
examine developer adoption rates and performance impact with ease.
• Because we can access the entire code base where a feature is
deployed, we can correct design mistakes with low cost. For example,
if a feature proves too confusing to use in practice, we can change
its syntax. Indeed, Jane Street regularly makes broad updates to its
code base, and this kind of change fits within our workflow. For
main-trunk OCaml, this means that features are more fully tested
than they could be otherwise.
Upstreaming – contributing back to an open-source project and working
in a language that reaches beyond our walls – remains a core value for
the compilers team. (Indeed, one of my explicit job responsibilities
at Jane Street is to help facilitate this process.) In addition, all
our compiler work is fully [open-source].
That said, we have a number of features we’re excited to share
progress on, listed below. Expect to see proper proposals for these
(posted to the [RFCs repo]) in due course.
*[Local allocations]*, implemented by Stephen Dolan and Leo White.
This adds support for stack-allocated blocks, enforcing memory safety
by requiring that heap-allocated blocks never point to stack-allocated
blocks (and stack-allocated blocks never point to shorter-lived
stack-allocated blocks). Function arguments can be annotated as
`local_' to say that the function does not store that argument. For
example,
┌────
│ val map : 'a list -> f:local_ ('a -> 'b) -> 'b list
└────
says that `map' does not store the function it is passed anywhere in
the output, allowing the function closure to be stack-allocated.
Stack-allocated record, variants, etc., are also possible. See also
[Stephen Dolan’s talk at ML’22].
This is a large addition to OCaml’s type system, and we’re still
learning about how to use it best. We are actively learning from its
deployment within Jane Street to influence the final version of this
feature for upstreaming.
*`include functor'*, [implemented] by Chris Casinghino. This syntactic
extension allows a module to include the results of applying a functor
to the prefix of the module already written. For example:
┌────
│ module type Indexed_collection = sig
│ type 'a t
│ val mapi : 'a t -> f:(int -> 'a -> 'b) -> 'b t
│ end
│
│ module Make_map(M : Indexed_collection) : sig
│ val map : 'a M.t -> f:('a -> 'b) -> 'b M.t
│ end = struct
│ let map t ~f = M.mapi t ~f:(fun _ -> f)
│ end
│
│ module List = struct
│ type 'a t =
│ | Nil
│ | Cons of 'a * 'a t
│
│ let mapi t ~f = ...
│
│ include functor Make_map
│ end
│
│ let evens = List.(map (Cons (1, Cons (2, Cons (3, Nil)))) ~f:(fun x -> x*2))
└────
Despite being “just” a syntactic convenience, this has already
received wide uptake within Jane Street.
*Comprehensions*, [implemented] by Antal Spector-Zabusky. This adds
support for both list and array comprehensions, such as
┌────
│ let pythagorean_triples_up_to n =
│ [ a,b,c
│ for a = 1 to n
│ for b = a to n
│ for c = b to n
│ when a * a + b * b = c * c ]
└────
The feature works similarly for arrays, with `[| ... |]' syntax.
*Immutable arrays*, [implemented] by Antal Spector-Zabusky. This adds
support for immutable arrays. These are just like normal arrays, but
immutability ensures that the contents of the array do not change.
*Unboxed types*, with a very early [implementation] by Chris
Casinghino and [proposal] by me (with the help of my colleagues).
[Stephen Dolan] [and I] have given talks about the design.
Currently, all variables and fields must store values that are stored
in a single machine word (e.g. 64 bits); furthermore, this word must
either be a pointer to garbage-collected memory or have its bottom bit
tagged to denote that the garbage collector should skip it. Unboxed
types relax this restriction, allowing a single variable or field to
hold structures smaller or larger than a word, or store a word that
the garbage collector will know not to scan. In so doing, unboxed
types effectively allow records and variants to be inlined into one
another, enabling programmers to structure their compile-time
abstractions differently than their run-time memory layout.
The core innovation is the notion of a /layout/, which classifies a
type (much like a type classifies a value). A layout describes how a
value should be stored and whether it can be examined by the garbage
collector. By assigning layouts to abstract types and type variables,
we can abstract over non-word layouts. Much more is in the [proposal].
*Polymorphic parameters*, [implemented] and [documented] by Leo White.
This feature allows function parameters to have polymorphic types. For
example:
┌────
│ let select
│ : 'b 'c. selector:('a. ('a * 'a) -> 'a) -> ('b * 'b) list -> ('c * 'c) list -> ('b * 'c) list
│ = fun ~selector bbs ccs ->
│ List.map2 (fun bb cc -> selector bb, selector cc) bbs ccs
└────
The `select' function chooses either the first components or the
second components of the input lists to comprise the output list. The
`selector' function must be polymorphic, because the elements in the
pairs in the input lists may have different types. (Note that `'b' and
`'c' are universally quantified in the type of `select', so we know
that the type checker does not unify them.)
[open-source] <https://github.com/ocaml-flambda/ocaml-jst>
[RFCs repo] <https://github.com/ocaml/RFCs>
[Local allocations]
<https://github.com/ocaml-flambda/ocaml-jst/tree/main/jane/doc>
[Stephen Dolan’s talk at ML’22]
<https://www.youtube.com/watch?v=yGRn5ZIbEW8>
[implemented] <https://github.com/ocaml-flambda/ocaml-jst/pull/32>
[implemented] <https://github.com/ocaml-flambda/ocaml-jst/pull/46>
[implemented] <https://github.com/ocaml-flambda/ocaml-jst/pull/47>
[implementation] <https://github.com/ocaml-flambda/ocaml-jst/pull/48>
[proposal] <https://github.com/ocaml/RFCs/pull/34>
[Stephen Dolan] <https://www.youtube.com/watch?v=RV-4Xddk0Yc>
[and I] <https://www.youtube.com/watch?v=Vevld4cXSYk>
[implemented] <https://github.com/ocaml-flambda/ocaml-jst/pull/14>
[documented]
<https://github.com/lpw25/ocaml-jst/blob/poly-param2/jane/doc/polymorphic-parameters.md>
diskuvbox: small set of cross-platform CLI tools
════════════════════════════════════════════════
Archive:
<https://discuss.ocaml.org/t/ann-diskuvbox-small-set-of-cross-platform-cli-tools/9663/8>
jbeckford announced
───────────────────
Version 0.1.1 is now available from opam. Upgrade with:
┌────
│ opam update
│ opam upgrade diskuvbox
└────
Changes include:
• A bugfix for 32-bit OCaml systems; copying of large files now works.
• This is the first version to officially include standalone binaries.
They are available at
<https://github.com/diskuv/diskuvbox/releases/tag/0.1.1> for
Linux/Intel 32- and 64-bit, macOS/Intel, macOS/Silicon,
Windows/Intel 32- and 64-bit
http_async 0.2.0
════════════════
Archive: <https://discuss.ocaml.org/t/ann-http-async-0-2-0/10784/1>
Anurag Soni announced
─────────────────────
I’d like to announce the initial release of [http_async].
*Http_async*: provides a HTTP/1.1 web server for [async].
The library started life as a wrapper around httpaf, but has evolved
over time to be a standalone implementation of an HTTP/1.1 server. It
implements its own parser that works on big strings and is reasonably
fast. The library supports the common use-cases of connection
pipelining, streaming request and response bodies. Servers can be run
on both IP and Unix domain sockets, and ships with optional
`Core.Command' entry points.
The implementation initially started as a testing ground for my I/o
library [shuttle], but I’ve been happy enough with the resulting api
that I intend to support this as a standalone library moving forward.
As far as performance is concerned, I always recommend doing your own
tests with workloads that are closer to the intended use, but in my
tests the performance seems very reasonable. There is a version of
http_async that’s part of the http benchmarks run for the multicore
project and the results looked fairly decent when compared to other
single threaded runtimes (see
<https://github.com/ocaml-multicore/retro-httpaf-bench/pull/24> for
more details)
[http_async] <https://github.com/anuragsoni/http_async>
[async] <https://opensource.janestreet.com/async/>
[shuttle] <https://github.com/anuragsoni/shuttle>
Hello World server
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
┌────
│ open! Core
│ open Async
│ open Http_async
│
│ let () =
│ Command_unix.run
│ (Server.run_command ~summary:"Hello world HTTP Server" (fun peer_addr _request ->
│ return (Response.create `Ok, Body.Writer.string "Hello World")))
│ ;;
└────
The library doesn’t ship with any routers, or a middleware composition
API with the intention that those will be layers that will live as a
separate library. It should be easy to plug this into a third party
routing library like `ocaml-dispatch' or `routes'. Some examples can
be seen at:
<https://github.com/anuragsoni/http_async/tree/main/example> and
<https://github.com/anuragsoni/http_async/blob/main/bench/server_bench.ml>
and documentation is available [online].
[online]
<https://anuragsoni.github.io/http_async/http_async/Http_async/index.html>
Install
╌╌╌╌╌╌╌
◊ To use the version published on opam:
┌────
│ opam install http_async
└────
◊ For the development version:
┌────
│ opam pin add http_async.dev git+https://github.com/anuragsoni/http_async.git
└────
If you try the library and experience any issues, or have further
questions, please report an issue on the Github Issue tracker.
Other OCaml News
════════════════
>From the ocaml.org blog
───────────────────────
Here are links from many OCaml blogs aggregated at [the ocaml.org
blog].
• [How to get started with OCaml in 2022]
[the ocaml.org blog] <https://ocaml.org/blog/>
[How to get started with OCaml in 2022]
<https://tech.ahrefs.com/how-to-get-started-with-ocaml-in-2022-2f22b578b984>
Old CWN
═══════
If you happen to miss a CWN, you can [send me a message] and I’ll mail
it to you, or go take a look at [the archive] or the [RSS feed of the
archives].
If you also wish to receive it every week by mail, you may subscribe
[online].
[Alan Schmitt]
[send me a message] <mailto:alan.schmitt at polytechnique.org>
[the archive] <https://alan.petitepomme.net/cwn/>
[RSS feed of the archives] <https://alan.petitepomme.net/cwn/cwn.rss>
[online] <http://lists.idyll.org/listinfo/caml-news-weekly/>
[Alan Schmitt] <https://alan.petitepomme.net/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/caml-news-weekly/attachments/20221115/4aef1f82/attachment-0001.html>
More information about the caml-news-weekly
mailing list