[cwn] Attn: Development Editor, Latest OCaml Weekly News
Alan Schmitt
alan.schmitt at polytechnique.org
Tue Jun 15 06:29:40 PDT 2021
Hello
Here is the latest OCaml Weekly News, for the week of June 08 to 15,
2021.
Table of Contents
─────────────────
Dependency visualization tool
Multicore OCaml: May 2021
First release of Popper
Orsetto: structured data interchange languages (release 1.1)
Old CWN
Dependency visualization tool
═════════════════════════════
Archive:
<https://discuss.ocaml.org/t/dependency-visualization-tool/7964/2>
Chet Murthy announced
─────────────────────
In the past, I've found doing this sort of dependency-graphing to be
very useful for figuring out where modularization in a large system
needs to be applied. And it's not very hard to do, based on ocamldep
output or other dependency information that already exists.
On a lark, I wrote something that did this for installed findlib
packages: <https://github.com/chetmurthy/not-ocamlfind#package-graph>
I mention this not to plug the tool, but rather, just to note that
it's really easy, and yeah, people should avail themselves of this
more.
Multicore OCaml: May 2021
═════════════════════════
Archive: <https://discuss.ocaml.org/t/multicore-ocaml-may-2021/7990/1>
Anil Madhavapeddy announced
───────────────────────────
Welcome to the May 2021 [Multicore OCaml] monthly report! This month's
update along with the [previous updates] have been compiled by @avsm,
@ctk21, @kayceesrk and @shakthimaan.
Firstly, all of our upstream activity on the OCaml compiler is now
reported as part of the shiny new [compiler development newsletter #2]
that @gasche has started. This represents a small but important shift
– domains-only multicore is firmly locked in on the upstream roadmap
for OCaml 5.0 and the whole OCaml compiler team has been helping and
contributing to it, with the [GC safe points] feature being one of the
last major multicore-prerequisites (and due to be in OCaml 4.13 soon).
This multicore newsletter will now focus on getting our ecosystem
ready for domains-only multicore in OCaml 5.0, and on how the
(not-yet-official) effect system and multicore IO stack is
progressing. It's a long one this month, so settle in with your
favourite beverage and let's begin :-)
[Multicore OCaml] <https://github.com/ocaml-multicore/ocaml-multicore>
[previous updates] <https://discuss.ocaml.org/tag/multicore-monthly>
[compiler development newsletter #2]
<https://discuss.ocaml.org/t/ocaml-compiler-development-newsletter-issue-2-may-2021/7965>
[GC safe points] <https://github.com/ocaml/ocaml/pull/10039>
OCaml Multicore: 4.12.0+domains
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
The multicore compiler now supports CTF runtime traces of its garbage
collector and there are [tools to display chrome tracing
visualisations] of the garbage collector events. A number of
performance improvements (see speedup graphs later on) that highlight
some ways to make best use of multicore were made to the existing
benchmarks in Sandmark. There has also been work on scaling up to 128
cores/domains for task-based parallelism in domainslib using [work
stealing deques], bringing us closer to Cilk-style task-parallel
performance.
As important as new features are what we have decided _not_ to
do. We've been working on and evaluating Domain Local Allocation
Buffers (DLABs) for some time, with the intention of reducing the cost
of minor GCs. We've found that the resulting performance didn't match
our expectations (_vs_ the complexity of the change), and so we've
decided not to proceed with this for OCaml 5.0. You can find the
[DLAB summary] page summarises our experiences. We'll come back to
this post-OCaml 5.0 when there are fewer moving parts.
[tools to display chrome tracing visualisations]
<https://github.com/ocaml-multicore/eventlog-tools/tree/multicore_wip>
[work stealing deques]
<https://github.com/ocaml-multicore/domainslib/pull/29>
[DLAB summary]
<https://github.com/ocaml-multicore/ocaml-multicore/wiki/Domain-Local-Allocation-Buffers-Addendum>
Ecosystem changes to prepare for 5.0.0 domains-only
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
As we are preparing 5.0 branches with the multicore branches over the
coming months, we are stepping up preparations to ensure the OCaml
ecosystem is ready.
Making the multicore compilers available by default in opam-repo
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
Over the next few week, we will be merging the multicore
4.12.0+domains and associated packages from their opam remote over in
[ocaml-multicore/multicore-opam] into the mainline
opam-repository. This is to make it more convenient to use the variant
compilers to start testing your own packages with ~Domain~s.
As part of this change, there are two new base packages that will be
available in opam-repository:
• `base-domains': This package indicates that the current compiler has
the `Domain' module.
• `base-effects': This package indicates the current compiler has the
experimental effect system.
By adding a dependency on these packages, the only valid solutions
will be `4.12.0+domains' (until OCaml 5.0 which will have this module)
or `4.12.0+effects'.
The goal of this is to let community packages more easily release
versions of their code using Domains-only parallelism ahead of OCaml
5.0, so that we can start migration and thread-safety early. We do
not encourage anyone to take a dependency on base-effects currently,
as it is very much a moving target.
This opam-repository change isn't in yet, but I'll comment on this
post when it is merged.
[ocaml-multicore/multicore-opam]
<https://github.com/ocaml-multicore/multicore-opam>
Adapting the Stdlib for thread-safety
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
One of the first things we have to do before porting third-party
libraries is to get the Stdlib ready for thread-safety. This isn't
quite as simple as it might appear at first glance: if we adopt the
naïve approach of simply putting a mutex around every bit of global
state, our sequential performance will slow down. Therefore we are
performing a more fine-grained analysis and fixes, which can be seen
[on the multicore stdlib page].
For anyone wishing to contribute: hunt through the Stdlib for global
state, and categorise it appropriately, and then create a test case
exercising that module with multiple Domains running, and submit a PR
to [ocaml-multicore]. In general, if you see any build failures or
runtime failures now, we'd really appreciate an issue being filed
there too. You can see some good examples of such issues [here] (for
mirage-crypto) and [here] (for Coqt).
[on the multicore stdlib page]
<https://github.com/ocaml-multicore/ocaml-multicore/wiki/Safety-of-Stdlib-under-Multicore-OCaml>
[ocaml-multicore] <https://github.com/ocaml-multicore/ocaml-multicore>
[here] <https://github.com/ocaml-multicore/ocaml-multicore/issues/574>
[here] <https://github.com/ocaml-multicore/ocaml-multicore/issues/568>
Porting third-party libraries to Domains
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
As I mentioned last month, we put a call out for libraries and
maintainers who wanted to port their code over. We're starting with
the following libraries and applications this month:
• *Lwt*: the famous lightweight-threads library now has a PR to add
[Lwt_domains]. This is the first simple(ish) step to using
multicore cores with Lwt: it lets you run a pure (non-Lwt) function
in another Domain via `detach : ('a -> 'b) -> 'a -> 'b Lwt.t'.
• *Mirage-Crypto*: the next library we are adapting is the
cryptography library, since it is also low-hanging fruit that
should be easy to parallelise (since crypto functions do not have
much global state). The port is still ongoing, as there are some
minor build failures and also Stdlib functions in Format that
aren't yet thread-safe that are [causing failures].
• *Tezos-Node*: the bigger application we are applying some of the
previous dependencies too is Tezos-Node, which makes use of the
dependency chain here via Lwt, mirage-crypto, Irmin, Cohttp and
many other libraries. We've got this [compiling under
4.12.0+domains] now and mostly passing the test suite, but will
only report significant results once the dependencies and Stdlib
are passing.
• *Owl*: OCaml's favourite machine learning library works surprisingly
well out-of-the-box with 4.12.0+domains. An experiment for a
significant machine-learning codebase written using it saw about a
2-4x speedup before some false-sharing bottlenecks kicked in. This
is pretty good going given that we made no changes to the codebase
itself, but stay tuned for more improvements over the coming months
as we analyse the bottleneck.
This is hopefully a signal to all of you to start "having a go" with
4.12.0+domains on your own applications, and particularly with respect
to seeing how wrapping it in Domains works out and identifying global
state. You can read our handy [tutorial on parallel programming with
Multicore OCaml].
We are developing some tools to help find global state, but we're
going to all need to work together to identify some of these cases and
begin migration. Crucially, we need some diversity in our dependency
chains – if you have interesting applications using (e.g.) Async or
the vanilla `Thread' module and have some cycles to work with us,
please get in touch with me or @kayceesrk .
[Lwt_domains] <https://github.com/ocsigen/lwt/pull/860>
[causing failures]
<https://github.com/ocaml-multicore/ocaml-multicore/issues/563>
[compiling under 4.12.0+domains]
<https://gitlab.com/tezos/tezos/-/merge_requests/2671>
[tutorial on parallel programming with Multicore OCaml]
<https://github.com/ocaml-multicore/parallel-programming-in-multicore-ocaml>
4.12.0+effects
╌╌╌╌╌╌╌╌╌╌╌╌╌╌
The effects-based [eio library] is coming together nicely, and the
interface and design rationales are all up-to-date in the README of
the repository. The primary IO backend is [ocaml-uring], which we are
preparing for a separate release to opam-repository now as it also
works fine on the sequential runtime for Linux (as long as you have a
fairly recent kernel. Otherwise the kernel crashes). We also have a
[Grand Central Dispatch effect backend] to give us a totally different
execution model to exercise our effect handler abstractions.
While we won't publish the performance numbers for the effect-based IO
this month, you can get a sense of the sorts of tests we are running
by looking at the [retro-httpaf-bench] repository, which now has
various permutations of effects-based, uring-based and select-based
webservers. We've submitted a talk to the upcoming OCaml Workshop
later this summer, which, if accepted, will give you a deepdive into
our effect-based IO.
As always, we begin with the Multicore OCaml ongoing and completed
tasks. The ecosystem improvements are then listed followed by the
updates to the Sandmark benchmarking project. Finally, the upstream
OCaml work is mentioned for your reference. For those of you that
have read this far and can think of nothing more fun than hacking on
multicore programming runtimes, we are hiring in the UK, France and
India – please find the job postings at the end!
[eio library] <https://github.com/ocaml-multicore/eio>
[ocaml-uring] <https://github.com/ocaml-multicore/ocaml-uring>
[Grand Central Dispatch effect backend]
<https://github.com/ocaml-multicore/eio/pull/26>
[retro-httpaf-bench]
<https://github.com/ocaml-multicore/retro-httpaf-bench>
Multicore OCaml
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
Ongoing
┄┄┄┄┄┄┄
• [ocaml-multicore/ocaml-multicore#552] Add a
force_instrumented_runtime option to configure
A new `--enable-force-instrumented-runtime' option is introduced to
facilitate use of the instrumented runtime on linker invocations to
obtain event logs.
• [ocaml-multicore/ocaml-multicore#553] Testsuite failures with
flambda enabled
A list of tests are failing on `b23a416' with flambda enabled, and
they need to be investigated further.
• [ocaml-multicore/ocaml-multicore#555] runtime: CAML_TRACE_VERSION is
now set to a Multicore specific value
Define a `CAML_TRACE_VERSION' to distinguish between Multicore OCaml
and trunk for the runtime.
• [ocaml-multicore/ocaml-multicore#558] Refactor Domain.{spawn/join}
to use no critical sections
The PR removes the use of `Domain.wait' and critical sections in
`Domain.{spawn/join}'.
• [ocaml-multicore/ocaml-multicore#559] Improve the Multicore GC Stats
A draft PR to include more Multicore GC statistics when using
`OCAMLRUNPARAM=v=0x400'.
[ocaml-multicore/ocaml-multicore#552]
<https://github.com/ocaml-multicore/ocaml-multicore/pull/552>
[ocaml-multicore/ocaml-multicore#553]
<https://github.com/ocaml-multicore/ocaml-multicore/issues/553>
[ocaml-multicore/ocaml-multicore#555]
<https://github.com/ocaml-multicore/ocaml-multicore/pull/555>
[ocaml-multicore/ocaml-multicore#558]
<https://github.com/ocaml-multicore/ocaml-multicore/pull/558>
[ocaml-multicore/ocaml-multicore#559]
<https://github.com/ocaml-multicore/ocaml-multicore/pull/559>
Completed
┄┄┄┄┄┄┄┄┄
• [ocaml-multicore/ocaml-multicore#508] Domain Local Allocation
Buffers
The Domain Local Allocation Buffer implementation for OCaml
Multicore has been dropped for now. A discussion is on the PR itself
and there is a wiki page [here].
• [ocaml-multicore/ocaml-multicore#527] Port eventlog to CTF
The porting of the `eventlog' implementation to the Common Trace
Format is now complete.
For an introduction to producing Chrome trace visualizations of the
runtime events see [eventlog-tools]. This postprocessing tool turns
the CTF trace into the Chrome tracing format that allows interactive
visualizations like this:
<https://aws1.discourse-cdn.com/standard11/uploads/ocaml/optimized/2X/7/7984354f12695102e97bd6a37202ea8da9a6decf_2_1380x950.jpeg>
• [ocaml-multicore/ocaml-multicore#543] Parallel version of
weaklifetime test
A parallel version of the `weaklifetime.ml' test is now added to the
test suite.
• [ocaml-multicore/ocaml-multicore#546] Coverage of domain life-cycle
in domain_dls and ephetest_par tests
Additional tests to increase test coverage for domain life-cycle for
`domain_dls.ml' and `ephetest_par.ml'.
• [ocaml-multicore/ocaml-multicore$#550] Lazy effects test
Inclusion of a test to address effects with Lazy computations for a
number of different use cases.
• [ocaml-multicore/ocaml-multicore#557] Remove unused domain functions
A clean-up to remove unused functions in `domain.c' and `domain.h'.
[ocaml-multicore/ocaml-multicore#508]
<https://github.com/ocaml-multicore/ocaml-multicore/pull/508>
[here]
<https://github.com/ocaml-multicore/ocaml-multicore/wiki/Domain-Local-Allocation-Buffers-Addendum>
[ocaml-multicore/ocaml-multicore#527]
<https://github.com/ocaml-multicore/ocaml-multicore/pull/527>
[eventlog-tools]
<https://github.com/ocaml-multicore/eventlog-tools/tree/multicore_wip>
[ocaml-multicore/ocaml-multicore#543]
<https://github.com/ocaml-multicore/ocaml-multicore/pull/543>
[ocaml-multicore/ocaml-multicore#546]
<https://github.com/ocaml-multicore/ocaml-multicore/pull/546>
[ocaml-multicore/ocaml-multicore$#550]
<https://github.com/ocaml-multicore/ocaml-multicore/pull/550>
[ocaml-multicore/ocaml-multicore#557]
<https://github.com/ocaml-multicore/ocaml-multicore/pull/557>
Ecosystem
╌╌╌╌╌╌╌╌╌
Ongoing
┄┄┄┄┄┄┄
• [ocaml-multicore/eventlog-tools#2] Add a pausetimes tool
The `eventlog_pausetimes' tool takes a directory of eventlog files
and computes the mean, max pause times, as well as the distribution
up to the 99.9th percentiles. For example:
┌────
│ ocaml-eventlog-pausetimes /home/engil/dev/ocaml-multicore/trace3/caml-426094-* name
│ {
│ "name": "name",
│ "mean_latency": 718617,
│ "max_latency": 33839379,
│ "distr_latency": [191,250,707,16886,55829,105386,249272,552640,1325621,13312993,26227671]
│ }
└────
• [domainslib#29] Task stealing with CL deques
This ongoing work to use task-stealing Chase Lev deques for
scheduling tasks across domains is looking very
promising. Particularly for machines with 128 cores.
* [ocaml-multicore/retro-httpaf-bench#10] Add Eio benchmark
The addition of an Eio benchmark for retro-httpaf-bench. This is a
work-in-progress.
• [ocaml-multicore/eio#26] Grand Central Dispatch Backend
An early draft PR that implements the Grand Central Dispatch (GCD)
backend for Eio.
• [ocsigen/lwt#860] Lwt_domain: An interfacet to Multicore parallelism
An on-going effort to introduce `Lwt_domain' for performing
computations to CPU cores using Multicore OCaml's Domains.
[ocaml-multicore/eventlog-tools#2]
<https://github.com/ocaml-multicore/eventlog-tools/pull/2>
[domainslib#29] <https://github.com/ocaml-multicore/domainslib/pull/29>
[ocaml-multicore/retro-httpaf-bench#10]
<https://github.com/ocaml-multicore/retro-httpaf-bench/pull/10>
[ocaml-multicore/eio#26]
<https://github.com/ocaml-multicore/eio/pull/26>
[ocsigen/lwt#860] <https://github.com/ocsigen/lwt/pull/860>
Completed
┄┄┄┄┄┄┄┄┄
retro-httpaf-bench
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
The `retro-httpaf-bench' repository contains scripts for running HTTP
server benchmarks.
• [ocaml-multicore/retro-httpaf-bench#6] Move OCaml to 4.12
The build scripts have been updated to use 4.12.0.
• [ocaml-multicore/retro-httpaf-bench#8] Adds a Rust benchmark using
hyper
The inclusion of the Hyper benchmark limited to a single core to
match the other existing benchmarks.
• [ocaml-multicore/retro-httpaf-bench#9] Release builds for dune,
stretch request volumes, rust fixes and remove mimalloc
The Dockerfile, README, build_benchmarks.sh and run_benchmarks.sh
files have been updated.
• [ocaml-multicore/retro-httpaf-bench#15] Make benchmark more
realistic
The PR enhances the implementation to correctly simulate a
hypothetical database request, and the effects code has been updated
accordingly.
[ocaml-multicore/retro-httpaf-bench#6]
<https://github.com/ocaml-multicore/retro-httpaf-bench/pull/6>
[ocaml-multicore/retro-httpaf-bench#8]
<https://github.com/ocaml-multicore/retro-httpaf-bench/pull/8>
[ocaml-multicore/retro-httpaf-bench#9]
<https://github.com/ocaml-multicore/retro-httpaf-bench/pull/9>
[ocaml-multicore/retro-httpaf-bench#15]
<https://github.com/ocaml-multicore/retro-httpaf-bench/pull/5>
eio
┈┈┈
The `eio' library provides an effects-based parallel IO stack for
Multicore OCaml.
• [ocaml-multicore/eio#18] Add fibreslib library
The `promise' library has been renamed to `fibreslib' to avoid
naming conflict with the existing package in opam, and the API
(waiters and effects) has been split into its own respective
modules.
• [ocaml-multicore/eio#19] Update to latest ocaml-uring
The code and configuration files have been updated to use the latest
`ocaml-uring'.
• [ocaml-multicore/eio#20] Add Fibreslib.Semaphore
Implemented the `Fibreslib.Semaphone' module that is useful for
rate-limiting, and based on OCaml's `Semaphore.Counting'.
• [ocaml-multicore/eio#21] Add high-level Eio API
A new Eio library with interfaces for sources and sinks. The README
documentation has been updated with motivation and usage.
• [ocaml-multicore/eio#22] Add switches for structured concurrency
Implementation of structured concurrency with documentation examples
for tracing and testing with mocks.
• [ocaml-multicore/eio#23] Rename repository to eio
The Effects based parallel IO for OCaml repository has now been
renamed from `eioio' to `eio'.
• [ocaml-multicore/eio#24] Rename lib_eioio to lib_eunix
The names have been updated to match the dune file.
• [ocaml-multicore/eio#25] Detect deadlocks
An exception is now raised to detect deadlocks if the scheduler
finishes while the main thread continues to run.
• [ocaml-multicore/eio#27] Convert expect tests to MDX
The expected tests have been updated to use the MDX format, and this
avoids the need for ppx libraries.
• [ocaml-multicore/eio#28] Use splice to copy if possible
The effect Splice has been implemented along with the update to
ocaml-uring, and necessary documentation.
• [ocaml-multicore/eio#29] Improve exception handling in switches
Additional exception checks to handle when multiple threads fail,
and for `Switch.check' and `Fibre.fork_ignore'.
• [ocaml-multicore/eio#30] Add eio_main library to select backend
automatically
Use `eio_main' to select the appropriate backend (`eunix', for
example) based on the platform.
• [ocaml-multicore/eio#31] Add Eio.Flow API
Implemented a Flow module that allows combinations such as
bidirectional flows and closable flows.
• [ocaml-multicore/eio#32] Initial support for networks
Eio provides a high-level API for networking, and the `Network'
module has been added.
• [ocaml-multicore/eio#33] Add some design rationale notes to the
README
The README has been updated with design notes, and reference to
further reading on the principles of Object-capability model.
• [ocaml-multicore/eio#34] Add shutdown, allow closing listening
sockets, add cstruct_source
Added cstruct_source, `shutdown' method along with source, sink and
file descriptor types.
• [ocaml-multicore/eio#35] Add Switch.on_release to auto-close FDs
We can now attach resources such as file descriptors to switches,
and these are freed when the the switch is finished.
[ocaml-multicore/eio#18]
<https://github.com/ocaml-multicore/eio/pull/18>
[ocaml-multicore/eio#19]
<https://github.com/ocaml-multicore/eio/pull/19>
[ocaml-multicore/eio#20]
<https://github.com/ocaml-multicore/eio/pull/20>
[ocaml-multicore/eio#21]
<https://github.com/ocaml-multicore/eio/pull/21>
[ocaml-multicore/eio#22]
<https://github.com/ocaml-multicore/eio/pull/22>
[ocaml-multicore/eio#23]
<https://github.com/ocaml-multicore/eio/pull/23>
[ocaml-multicore/eio#24]
<https://github.com/ocaml-multicore/eio/pull/24>
[ocaml-multicore/eio#25]
<https://github.com/ocaml-multicore/eio/pull/25>
[ocaml-multicore/eio#27]
<https://github.com/ocaml-multicore/eio/pull/27>
[ocaml-multicore/eio#28]
<https://github.com/ocaml-multicore/eio/pull/28>
[ocaml-multicore/eio#29]
<https://github.com/ocaml-multicore/eio/pull/29>
[ocaml-multicore/eio#30]
<https://github.com/ocaml-multicore/eio/pull/30>
[ocaml-multicore/eio#31]
<https://github.com/ocaml-multicore/eio/pull/31>
[ocaml-multicore/eio#32]
<https://github.com/ocaml-multicore/eio/pull/32>
[ocaml-multicore/eio#33]
<https://github.com/ocaml-multicore/eio/pull/33>
[ocaml-multicore/eio#34]
<https://github.com/ocaml-multicore/eio/pull/34>
[ocaml-multicore/eio#35]
<https://github.com/ocaml-multicore/eio/pull/35>
Sundries
┈┈┈┈┈┈┈┈
• [ocaml-multicore/domainslib#23] Running tests: moving to `dune
runtest' from manual commands in `run_test' target
The `dune runtest' command is now used to execute the tests.
• [ocaml-multicore/domainslib#24] Move to Mutex & Condition from
Domain.Sync.{notify/wait}
The channel implementation using `Mutex' and `Condition' is now
complete. The performance results are shown in the following graph:
<https://aws1.discourse-cdn.com/standard11/uploads/ocaml/optimized/2X/c/c3525f387ece415069bbac1f76525cf2ca96ef1d_2_930x1000.png>
• [ocaml-multicore/multicore-opam#53] Add base-domains and
base-effects packages
The `base-domains' and `base-effects' opam files have now been added
to multicore-opam.
• [ocaml-multicore/multicore-opam#54] Shift all multicore packages to
unique versions and base-domains dependencies
The naming convention is to now use `base-effects' and
`base-domains' everywhere.
[ocaml-multicore/domainslib#23]
<https://github.com/ocaml-multicore/domainslib/issues/23>
[ocaml-multicore/domainslib#24]
<https://github.com/ocaml-multicore/domainslib/pull/24>
[ocaml-multicore/multicore-opam#53]
<https://github.com/ocaml-multicore/multicore-opam/pull/53>
[ocaml-multicore/multicore-opam#54]
<https://github.com/ocaml-multicore/multicore-opam/pull/54>
Benchmarking
╌╌╌╌╌╌╌╌╌╌╌╌
Ongoing
┄┄┄┄┄┄┄
• [ocaml-bench/sandmark#230] Build for 4.13.0+trunk with dune.2.8.1
A work-in-progress to upgrade Sandmark to use dune.2.8.1 to build
4.13.0+trunk and generate the benchmarks. You can test the same
using:
┌────
│ TAG='"macro_bench"' make run_config_filtered.json
│ RUN_CONFIG_JSON=run_config_filtered.json make ocaml-versions/4.13.0+trunk.bench
└────
[ocaml-bench/sandmark#230]
<https://github.com/ocaml-bench/sandmark/pull/230>
Completed
┄┄┄┄┄┄┄┄┄
Sandmark
┈┈┈┈┈┈┈┈
◊ Performance
• [ocaml-bench/sandmark#221] Fix up decompress iterations of work
The use of `parallel_for', simplification of `data_to_compress' to
use `String.init', and fix to correctly count the amount of work
configured and done produces the following speed improvements:
<https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/4/49a4b0b3f2e8e7c89e23c91c5f5307cdaa160d5c.png>
<https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/4/49bfcda236b0a900565c73b37867a7d35565b266.png>
• [ocaml-bench/sandmark#223] A better floyd warshall
An improvement to the Floyd Warshall implementation that fixes the
random seed so that it is repeatable, and improves the pattern
matching.
<https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/4/4a89577052976a094a4813b194aca1ebfb316a73.png>
<https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/b/be902a7ddc891b318c42a0e625b3335fbfeba1d0.png>
<https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/c/cbc5f44ea2b594b7d644a1ced93407e545c0bdb1.png>
• [ocaml-bench/sandmark#224] Some improvements for matrix
multiplication
The `matrix_multiplication' and `matrix_multiplication_multicore'
code have been updated for easier maintenance, and results are
written only after summing the values.
<https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/a/ac194a6ecd26bdb6d9e602378ed4b44eb1f67d8e.png>
<https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/4/4ecdd0c82cd59f50fffd9078cd85b5bbcf740e09.png>
• [ocaml-bench/sandmark#225] Better Multicore EA Benchmark
The Evolutionary Algorithm now inserts a poll point into `fittest'
to improve the benchmark results.
<https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/6/613a8f51682c02da672d12232bd8b244c65ba6d7.png>
<https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/b/b135d1269f57af9df575ab51ef7c31583713b42c.png>
• [ocaml-bench/sandmark#226] Better scaling for mandelbrot6_multicore
The `mandelbrot6_multicore' scales well now with the use of
`parallel_for' as observed in the following graphs:
<https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/3/3ae484f3942252583abc4d44cebc0051adec2b32.png>
<https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/b/b7dbb656928988548ae9668c0d1edfe5186479b2.png>
<https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/2/241f9360e8b3fee78b6f5d95ed40f7dcff24f813.png>
• [ocaml-bench/sandmark#227] Improve nbody_multicore benchmark with
high core counts
The `energy' function is now parallelised with `parallel_for_reduce'
for larger core counts.
<https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/d/d5bb284534e0f32e24c7d8937afa4cac022ef79c.png>
<https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/b/be718f3e5cc676f44c827dfa56bdfa01e376e447.png>
• [ocaml-bench/sandmark#229] Improve game_of_life benchmarks
The hot functions are now inlined to improve the `game_of_life'
benchmarks, and we avoid initialising the temporary matrix with
random numbers.
<https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/5/50c1b7e799a4c71ee4585409d1eba575bf0748ed.png>
<https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/e/eb7b75ddaa314fa80908f12758f974223eb70490.png>
[ocaml-bench/sandmark#221]
<https://github.com/ocaml-bench/sandmark/pull/221>
[ocaml-bench/sandmark#223]
<https://github.com/ocaml-bench/sandmark/pull/223>
[ocaml-bench/sandmark#224]
<https://github.com/ocaml-bench/sandmark/pull/224>
[ocaml-bench/sandmark#225]
<https://github.com/ocaml-bench/sandmark/pull/225>
[ocaml-bench/sandmark#226]
<https://github.com/ocaml-bench/sandmark/pull/226>
[ocaml-bench/sandmark#227]
<https://github.com/ocaml-bench/sandmark/pull/227>
[ocaml-bench/sandmark#229]
<https://github.com/ocaml-bench/sandmark/pull/229>
◊ Sundries
• [ocaml-bench/sandmark#215] Remove Gc.promote_to from
treiber_stack.ml
The 4.12+domains and 4.12+domains+effects branches have
`Gc.promote_to' removed from the runtime.
• [ocaml-bench/sandmark#216] Add configs for 4.12.0+stock,
4.12.0+domains, 4.12.0+domains+effects
The ocaml-version configuration files for 4.12.0+stock,
4.12.0+domains, and 4.12.0+domains+effects have now been included to
Sandmark.
• [ocaml-bench/sandmark#220] Attempt to improve the OCAMLRUNPARAM
documentation
The README has been updated with more documentation on the use of
OCAMLRUNPARAM configuration when running the benchmarks.
• [ocaml-bench/sandmark#222] Deprecate 4.06.1 and 4.10.0 and upgrade
to 4.12.0
The 4.06.1, 4.10.0 ocaml-versions have been removed and the CI has
been updated to use 4.12.0 as the default version.
[ocaml-bench/sandmark#215]
<https://github.com/ocaml-bench/sandmark/pull/215>
[ocaml-bench/sandmark#216]
<https://github.com/ocaml-bench/sandmark/pull/216>
[ocaml-bench/sandmark#220]
<https://github.com/ocaml-bench/sandmark/pull/220>
[ocaml-bench/sandmark#222]
<https://github.com/ocaml-bench/sandmark/pull/222>
current-bench
┈┈┈┈┈┈┈┈┈┈┈┈┈
• [ocurrent/current-bench#103] Ability to set scale on UI to start at
0
The graph origins now start from `[0, y_max+delta]' for the y-axis
for better comparison.
• [ocurrent/current-bench#121] Use string representation for docker
cpu setting.
The `OCAML_BENCH_DOCKER_CPU' setting now switches from Integer to
String to support a range of CPUs for parallel execution.
[ocurrent/current-bench#103]
<https://github.com/ocurrent/current-bench/issues/103>
[ocurrent/current-bench#121]
<https://github.com/ocurrent/current-bench/pull/121>
OCaml
╌╌╌╌╌
Ongoing
┄┄┄┄┄┄┄
• [ocaml/ocaml#10039] Safepoints
The Sandmark benchmark runs to obtain the performance numbers for
the Safepoints PR for 4.13.0+trunk have been published. The PR is
ready to be merged.
[ocaml/ocaml#10039] <https://github.com/ocaml/ocaml/pull/10039>
Job Advertisements
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
• [Multicore OCaml Runtime Systems Engineer] OCaml Labs (UK), Tarides
(France) and Segfault Systems (India)
• [Benchmark Tooling Engineer] Tarides
Our thanks to all the OCaml users, developers and contributors in the
community for their continued support to the project. Stay safe!
[Multicore OCaml Runtime Systems Engineer]
<https://discuss.ocaml.org/t/runtime-systems-engineer-ocaml-labs-uk-tarides-fr-segfault-systems-in-remote/7959>
[Benchmark Tooling Engineer]
<https://tarides.com/jobs/benchmark-tooling-engineer>
Acronyms
╌╌╌╌╌╌╌╌
• AMD: Advanced Micro Devices
• API: Application Programming Interface
• CI: Continuous Integration
• CPU: Central Processing Unit
• CTF: Common Trace Format
• DLAB: Domain Local Allocation Buffer
• EA: Evolutionary Algorithm
• GC: Garbage Collector
• GCD: Grand Central Dispatch
• HTTP: Hypertext Transfer Protocol
• OPAM: OCaml Package Manager
• MVP: Minimal Viable Product
• PR: Pull Request
• TPS: Transactions Per Second
• UI: User Interface
First release of Popper
═══════════════════════
Archive:
<https://discuss.ocaml.org/t/ann-first-release-of-popper/7992/1>
Joel Bjornson announced
───────────────────────
I am happy to announce a first release of [Popper] — an OCaml testing
library for writing simple unit tests as well as property-based tests,
using a uniform API.
A few things that distinguishes [Popper] from other property-based
testing libraries:
1) Generic shrinking — when a property-based test fails, the test
function attempts to shrink the sample data. This works for any
data type.
2) Any invariants used when constructing samples for property-based
tests are also respected when shrinking.
3) There is a [ppx] for automatically deriving sample functions
(generators in QuickCheck style libs) for custom data types.
The library also supports bundling and nesting tests arbitrarily,
colorful output inspired by [Alcotest], logging facilities and timing
info.
Here are some starting points:
• The [source repo].
• A [getting started guide].
• The [API docs].
Below is an example of what [Popper] output looks like. It was
generated by [this code].
<https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/8/86ab1380a1c443170656bc937a872c725cf6b78a.png>
This is a first release and there are few work-in-progress items. For
instance:
• A CLI for filtering and sorting tests.
• An API for better blending with [Lwt] code bases.
• A better way for limiting shrinking times for failing test samples.
Any suggestions or other feedback is much appreciated!
[Popper] <https://github.com/jobjo/popper>
[ppx] <https://opam.ocaml.org/packages/ppx_deriving_popper/>
[Alcotest] <https://github.com/mirage/alcotest>
[source repo] <https://github.com/jobjo/popper>
[getting started guide]
<https://jobjo.github.io/popper/getting_started/>
[API docs] <https://jobjo.github.io/popper/api/>
[this code]
<https://gist.github.com/jobjo/32f1b7297a330dc16d7a7c9fb15008fe>
[Lwt] <https://github.com/ocsigen/lwt>
Orsetto: structured data interchange languages (release 1.1)
════════════════════════════════════════════════════════════
Archive:
<https://discuss.ocaml.org/t/ann-orsetto-structured-data-interchange-languages-release-1-1/7995/1>
james woodyatt announced
────────────────────────
Announcing the release to OPAM of [Orsetto 1.1], an update to a
personal project of mine not sponsored by my employer. Licensed with
BSD 2-Clause.
[Orsetto 1.1] <http://opam.ocaml.org/packages/orsetto/>
Q. What is Orsetto?
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
Aspires to do eventually for OCaml more or less what [Serde] has done
for Rust, i.e. to be a curated and self-contained collection of
structured data interchange languages with a cohesive and unified
model of serialization and deserialization.
Two interchange languages are currently supported: [JSON] and [CBOR].
[Serde] <https://serde.rs>
[JSON] <https://json.org>
[CBOR] <https://cbor.io>
Q. What is new in this release?
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
The change log for the release is here: [CHANGES.md]
• Drops support for OCaml < 4.08.
• Major improvements in test coverage.
• Many corrections for logic errors.
• Issues [database] migrated to Atlassian Jira Cloud.
• A few important usability improvements (see below).
Some usability improvements are meant to replace obsolescent
interfaces (now marked obsolescent with `@ocaml.deprecated'). A few
improvements to some interfaces with especially suboptimal design were
updated with breaking changes, but these are not expected to cause
major upgrade problems for anyone.
Some things have not changed:
• Still requires *ocamlfind* and does not build with *dune*.
• Still only supports JSON and CBOR.
[CHANGES.md] <https://bitbucket.org/jhw/orsetto/src/r1.1/CHANGES.md>
[database] <https://conjury.atlassian.net/browse/ORS>
Q. It looks incomplete. What are your plans for future development?
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
Yes, it's a personal project. I'd welcome opportunities to collaborate
with others who share my vision for the project. As long as it's just
me working on this, development will be somewhat slow. I have a lot of
projects. This is the only open source one.
• *Orsetto 1.0.3* is the previous release. It offered parsers and
emitter combinators for JSON and CBOR for OCaml >= 4.06.1
(including 4.13~alpha1). The quality of its JSON support is
adequate, and it scores well on the [nst/JSONTestSuite] tests. The
quality of its CBOR support is provisional, and not recommended.
• *Orsetto 1.1* is the current release. It adds generalized and
extensible structured data interchange models with specializations
for producing emitters and parsers for JSON and CBOR. The quality
of the CBOR support is much improved.
• *Orsetto 1.2* is the next planned release. It will drop interfaces
marked `@caml.deprecated' in the 1.1 release. It will also drop
support for OCaml < 4.10, and I'd like to stop depending on
*ocamlfind*. I hope to add a PPX for deriving parsers and emitters
from OCaml data type definitions. I might also consider one or more
new interchange languages— suggestions are heartily encouraged.
[nst/JSONTestSuite] <https://github.com/nst/JSONTestSuite>
Old CWN
═══════
If you happen to miss a CWN, you can [send me a message] and I'll mail
it to you, or go take a look at [the archive] or the [RSS feed of the
archives].
If you also wish to receive it every week by mail, you may subscribe
[online].
[Alan Schmitt]
[send me a message] <mailto:alan.schmitt at polytechnique.org>
[the archive] <https://alan.petitepomme.net/cwn/>
[RSS feed of the archives] <https://alan.petitepomme.net/cwn/cwn.rss>
[online] <http://lists.idyll.org/listinfo/caml-news-weekly/>
[Alan Schmitt] <https://alan.petitepomme.net/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/caml-news-weekly/attachments/20210615/cfecb79e/attachment-0001.html>
More information about the caml-news-weekly
mailing list