[cwn] Attn: Development Editor, Latest OCaml Weekly News
Alan Schmitt
alan.schmitt at polytechnique.org
Tue Dec 7 02:23:11 PST 2021
Hello
Here is the latest OCaml Weekly News, for the week of November 30 to
December 07, 2021.
Table of Contents
─────────────────
New release of Windows DKML with system compiler and easy Opam switch creation
New release: ProVerif 2.04
Liquidshop 2.0 on Jan. 23, 2022!
Announcing our new Outreachy interns
OUPS meetup december 2021
Multicore OCaml: November 2021 with results of code review
New release of Menhir (20211128)
Advent of Code 2021
"What is an Operating System?": Anil Madhavapeddy on Signals and Threads
Other OCaml News
Old CWN
New release of Windows DKML with system compiler and easy Opam switch creation
══════════════════════════════════════════════════════════════════════════════
Archive:
<https://discuss.ocaml.org/t/ann-new-release-of-windows-dkml-with-system-compiler-and-easy-opam-switch-creation/8918/1>
jbeckford announced
───────────────────
Version 0.3.0 of [Diskuv OCaml (DKML)] is out. DKML is a distribution
that, among other things, can do a scripted install of OCaml and a
Microsoft compiler on Windows. It is in preview right now.
In an earlier release `with-dkml' was introduced. Placing `with-dkml'
in front of most OCaml tools will let those tools transparently have
access to Unix binaries and the Microsoft compiler (examples: 1. `dune
build' -> `with-dkml dune build'; 2. `opam install' -> `with-dkml opam
install'; or 3. `with-dkml bash' opens a Unix environment in the
current Windows directory).
Now if you upgrade to 0.3.0 you get more:
• There is a new Opam plugin bundled that will create a Windows local
switch. You can just do `mkdir playground', `cd playground' and
`opam dkml init' to create a new switch configured with Windows code
repositories and the Microsoft compiler. Thanks to @kit-ty-kate for
writing the [opam-build] plugin which helped in understanding how
Opam plugins work, and @dbuenzli for [many OCaml packages] leveraged
in both the plugin and with-dkml.
• The 4.12.1 OCaml compiler and runtime are installed once to the
"system", which means that if you create an Opam local switch the
OCaml compiler+runtime do not have to be recompiled. The huge waste
of time doing recompilation is now gone. All credit goes to @dra27
who created scripts in Opam that do almost the same thing. *There
will be a generally available Windows OCaml system compiler sometime
next year (that is, you won't need DKML), so consider the DKML
version to be an early and temporary preview of the Windows system
compiler.*
Links:
• [Installation instructions for the latest version]. Copy and paste
one block of commands, press OK when prompted, and 2 hours later
your installation will have finished.
• [Release notes for all versions including **upgrade instructions**]
• [Send a bug report] and/or [Stay updated on Twitter]
Known issues:
• Installing from mainline China frequently errors out. A short term
fix is available at
[https://gitlab.com/diskuv/diskuv-ocaml/-/issues/6#note_726814601]
[Diskuv OCaml (DKML)] <https://diskuv.gitlab.io/diskuv-ocaml>
[opam-build] <https://github.com/kit-ty-kate/opam-build>
[many OCaml packages] <https://github.com/dbuenzli>
[Installation instructions for the latest version]
<https://diskuv.gitlab.io/diskuv-ocaml/index.html#installation-instructions>
[Release notes for all versions including **upgrade instructions**]
<https://gitlab.com/diskuv/diskuv-ocaml/-/releases>
[Send a bug report]
<https://gitlab.com/diskuv/diskuv-ocaml/-/issues/new?issuable_template=Bug>
[Stay updated on Twitter]
<https://twitter.com/intent/follow?screen_name=diskuv>
[https://gitlab.com/diskuv/diskuv-ocaml/-/issues/6#note_726814601]
<https://gitlab.com/diskuv/diskuv-ocaml/-/issues/6#note_726814601>
New release: ProVerif 2.04
══════════════════════════
Archive:
<https://discuss.ocaml.org/t/new-release-proverif-2-04/8924/1>
Bruno Blanchet announced
────────────────────────
Vincent Cheval and I are happy to announce a new release of ProVerif,
version 2.04. ProVerif is an automatic security protocol verifier
that relies on the symbolic model of cryptography.
ProVerif is available at [http://proverif.inria.fr] It is also
available via opam (see installation instructions in Section 1.4 of
the manual <http://proverif.inria.fr/manual.pdf> )
Version 2.04 is mainly a bugfix release but it also contains a change
that may require minor changes in your input files: nested comments
are supported, so if you open a comment several times and close it
once, this is now an error.
Changes:
• Improved optimisation transforming mess facts into attacker facts
when the channel is a public term and not only a public name.
• Allow nested comments in the input file. All comments must be
closed. ProVerif will raise an error otherwise.
• Allow "let x = t in" construct inside the declarations of reduc and
equation.
• Fixed bug in subsumption: removing attacker facts containing ground
public terms was not correct.
• Improved unification modulo to avoid stack overflow.
• Fixed a bug that would cause a type error when using a typeConverter
function and the "let x = t in" construct inside a "not"
declaration.
• Fixed bug: make sure clauses coming from <-> or <=> in Horn clause
front-ends have distinct variables.
[http://proverif.inria.fr] <http://proverif.inria.fr/>
Liquidshop 2.0 on Jan. 23, 2022!
════════════════════════════════
Archive:
<https://discuss.ocaml.org/t/ann-liquidshop-2-0-on-jan-23-2022/8931/1>
Romain Beauxis announced
────────────────────────
We are glad to announce the second edition of Liquidshop, the
Liquidsoap workshop, which will be held online on *January 23rd 2022*.
To goal is to gather all people interested in Liquidsoap (of course)
but also in related topics:
• audio and video streaming
• radio broadcasting
• signal processing
• audio and video synthesis
and so on using whichever software. This is thus the place to share to
the world your favorite or cutting edge technologies and projects!
We will have 3 different tracks for the event, namely
• *showcases*: short presentations about a website / radio / art
installation that you built using Liquidsoap or other related tools
• *tech talks*: in-depth presentation of a technology related to
Liquidsoap and streaming in general
• *workshops*: user-centered freeform discussions about your project
or issues around Liquidsoap and streaming
If you're interested to participate, whether it is as an attendee or a
presenter, make sure to register via our website at
<http://www.liquidsoap.info/liquidshop/2> or directly via the form
available at <https://forms.gle/EDFsfMCS4dNKnrxc9>
We are super excited for this event. We have already secured a couple
of interesting speakers and we would love to get to know the community
better, see what y'all are doing with liquidsoap and other related
projects, community radios, live video, weird installations, etc. and
meet with everyone.
Finally, if any of you have some specific topics to discuss and would
like to learn more about Liquidsoap, this will be a great place to
connect!
Announcing our new Outreachy interns
════════════════════════════════════
Archive:
<https://discuss.ocaml.org/t/announcing-our-new-outreachy-interns/8932/1>
Sonja Heinze announced
──────────────────────
We're super excited to have another three Outreachy interns in the
OCaml community starting next Monday. The respective projects they'll
work on are:
• @JiaeK will work on an opam package monitoring dashboard for
v3.ocaml.org, by integrating the opam health check service.
• @SaySayo will work on _Ocaml platform_, the official OCaml VScode
plugin; concretely, on supporting `.eml'-files (used in Dream
applications).
• @ayc9 will work on a bundle of standard derivers for
`Ppxlib.Deriving' à la `ppx_deriving.std'
What's also super nice is that with @shonfeder and @pkel, we have two
non-Tarides/OCLC folks among us on the mentoring side. They followed
our [call for support]. @shonfeder is co-mentoring one of the
projects and @pkel acts as an "Outreachy volunteer", i.e. a mentor
who's there for non-project specific questions.
We'll keep you posted about the internships, @gs0510, @jonludlam,
@Juloo, @pitag, @pkel, @shonfeder, @tmattio
P.S.: We also submitted a fourth project on developing an odoc tool
that would help highlighting the differences between versions of a
package, which unfortunately didn't receive any viable
application. That's ok though! It's always better to offer too much
than too little and we might be able to re-submit something similar
some other round.
[call for support]
<https://discuss.ocaml.org/t/become-an-outreachy-mentor-support-the-growth-and-diversity-of-the-ocaml-community/8213>
OUPS meetup december 2021
═════════════════════════
Archive:
<https://discuss.ocaml.org/t/oups-meetup-december-2021/8933/1>
zapashcanon announced
─────────────────────
(this is in french only as the talks will be in french it's probably
not relevant for english speakers)
Le prochain OUPS aura lieu le *jeudi 9 décembre* 2021. Le rendez-vous
est fixé à *19h* en *salle 15-16 101*.
Ce meetup aura lieu dans les locaux de l'[IRILL] au [4 place Jussieu,
75005 Paris] en *salle 101*. Pour accéder à la salle, il faut aller à
la colonne 15 ou 16, monter un étage et accéder au couloir 15-16. Voir
les [instructions d'accès].
*L'inscription est obligatoire* pour pouvoir accéder au meetup ! Votre
nom complet doit être disponible. Merci de vous [inscrire] *le
mercredi 8 décembre* au plus tard.
Le meetup consistera en trois exposés à la suite de quoi, les
traditionnels pot et pizza party ne pouvant avoir lieux à l'IRILL,
nous proposerons à ceux qui le souhaitent de se rendre au [Baker
Street Pub] pour prolonger les discussions si la situation le permet.
Il est important de noter que *nous aurons peut-être à annuler
l'événement*, auquel cas nous vous en informerons le plus rapidement
possible sur les différents canaux.
Le programme des exposés de cette édition est donné ci-dessous et nous
profitons de l'occasion pour rappeler que nous sommes toujours à la
recherche de /propositions/* d'exposés pour les meetups suivants. Si
vous souhaitez proposer un exposé, contactez-nous sur le [zulip OUPS].
[4 place Jussieu, 75005 Paris]
<https://www.openstreetmap.org/#map=19/48.84650/2.35457>
[instructions d'accès] <https://www.irill.org/pages/access.html>
[inscrire] <https://www.meetup.com/fr-FR/ocaml-paris/events/282434883/>
[Baker Street Pub] <https://www.openstreetmap.org/node/2298160176>
[zulip OUPS] <https://oups.zulipchat.com>
[Monolith] par [François Pottier]
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
In this talk, I will give a brief presentation and demo of
Monolith, a tool that helps apply random testing or fuzz
testing to an OCaml library. Monolith provides a rich
specification language, which allows the user to describe
her library's API, and an engine, which generates clients
of this API and executes them. This reduces the problem of
testing a library to the problem of testing a complete
program, one that is effectively addressed by
off-the-shelf fuzzers such as AFL.
[Monolith] <https://gitlab.inria.fr/fpottier/monolith>
[François Pottier] <https://pauillac.inria.fr/~fpottier>
Générer 340_000 molécules valides par seconde sur un seul cœur et à dos de chameau par [François Berenger]
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
La conception de molécules par ordinateur est un sujet qui
revient à la mode. Pour générer in-silico des molécules
aux propriétés optimisées, une fonction de score peut être
couplée à un générateur moléculaire. Nous allons montrer
une méthode simple qui permet de générer uniquement des
molecules valides, à haute fréquence. Le logiciel libre
[FASMIFRA] implémente cette méthode en OCaml.
[François Berenger] <https://github.com/UnixJunkie>
[FASMIFRA] <https://github.com/UnixJunkie/FASMIFRA>
Pourquoi écrire du C quand on peut faire pire en OCaml ? par Pierre Chambart
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
Durant cette présentation, nous vous emmènerons visiter
les profondeurs d'OCaml. Vous y découvrirez comment écrire
du code lorsque votre seul objectif est d'éviter les
allocations. À condition bien évidemment de ne pas
craindre la honte. Vous pourrez également apercevoir de
véritables morceaux de flambda et de futurs véritables
morceaux de flambda2.
Multicore OCaml: November 2021 with results of code review
══════════════════════════════════════════════════════════
Archive:
<https://discuss.ocaml.org/t/multicore-ocaml-november-2021-with-results-of-code-review/8934/1>
Anil Madhavapeddy announced
───────────────────────────
Welcome to the November 2021 [Multicore OCaml] monthly report! This
month's update along with the [previous updates] have been compiled by
me, @ctk21, @kayceesrk and @shakthimaan.
[Multicore OCaml] <https://github.com/ocaml-multicore/ocaml-multicore>
[previous updates] <https://discuss.ocaml.org/tag/multicore-monthly>
Core Team Code Review
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
In late November, the entire OCaml development team convened for a
week-long code review and decision taking session on the multicore
merge for OCaml 5.0. Due to the size of the patchset, we broke up the
designs and presentations in five working groups. Here's a summary of
how each conversation went. As always, these decisions are subject to
change from the core team as we discover issues, so please do not take
any crucial decisions for your downstream projects on these. Our goal
for publicising these is to hear about any corrections you might feel
that we need to take on the basis of additional data that you might
have from your own codebases.
For the purposes of brevity, we do not include the full minutes of the
developer meetings. Overall, the multicore patches were deemed to be
broadly sound and suitable, and we recorded the important decisions
and tasks:
• *Pre-MVP:* Tasks that need to be done before we make the PR to
ocaml/ocaml in the coming month.
• *Post-MVP for 5.00:* Tasks that need to be done on ocaml/ocaml
before 5.00 release. /These tasks will block the OCaml 5.00
release./
• *Post-5.00:* Future looking tasks after 5.00 is released in
early/mid-2021.
WG1: Garbage Collector
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
The multicore runtime alters the memory allocation and garbage
collector to support multiple parallel threads of OCaml execution. It
utilizes a stop-the-world parallel minor collector, a StreamFlow like
multithreaded allocator and a mostly-concurrent major collector.
WG1 decided that compaction will not be in the 5.0 initial release, as
our best fit allocator has shown that a good memalloc strategy
obviates the need for expensive compaction. Of course, the multicore
memory allocator is different from bestfit, so we are in need of
community input to ensure our hypothesis involving not requiring
compaction is sound. If you do see such a use case of your application
heap becoming very fragmented when 5.0 is in beta, please get in
touch.
Pre-MVP
┈┈┈┈┈┈┈
• remove any traces of no-naked-pointers checker as it is irrelevant
in the pagetable-less multicore runtime.
• running `make parallel' for the testsuite should work
• move from `assert' to `CAMLassert'
• How to do safepoints from C: add documentation on
`caml_process_pending_actions' and a testsuite case for long-running
C bindings to multicore
• adopt the ephemeron bucket interface and do the same thing as 4.x
OCaml trunk
• check and document that `NOT_MARKABLE' can be used for libraries
like ancient that want out of heap objects
• check that we document what type of GC stats we return (global vs
domain local) for the various stats
Post-MVP for 5.00
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
• mark stack overflow fix, which shouldn't affect most runtime
allocation profiles
Post-5.00
┈┈┈┈┈┈┈┈┈
• statmemprof implementation
• mark pre-fetching
• investigate alternative minor heap implementations which maintain
performance but cut virtual memory usage
WG2: Domains
┄┄┄┄┄┄┄┄┄┄┄┄
Each domain in multicore can execute a thread of OCaml in parallel
with other domains. Several additions are made to OCaml to spawn new
domains, join domains that are terminating and provide domain local
storage. There is a stdlib module `Domain' and the underlying runtime
domain structures. A significant simplification in recent months is
that the standard Mutex/Channel/Semaphore modules can be used instead
of lower-level synchronisation primitives that were formerly available
in `Domain'.
The challenge for the runtime structures is to accurately maintain the
set of domains that must take part in stop-the-world sections in the
presence of domain termination and spawning, as well as ensuring that
a domain services stop-the-world requests when the main mutator is in
a blocking call; this is handled using a *backup thread* signaled from
`caml_enter_blocking_section' / `caml_leave_blocking_section'.
The multicore OCaml memory model was discussed, and the right scheme
selected for arm64 (Table 5b from [the paper]). The local data race
freedom (LDRF) property was agreed to be a balanced and predictable
approach for a memory model for OCaml 5.0. We do likely need to depend
on >C11 compiler for relaxed atomics in OCaml 5.0, so this will mean
dropping Windows MSVC support for the MVP (but mingw will work).
[the paper] <https://anil.recoil.org/papers/2018-pldi-memorymodel.pdf>
Pre-MVP
┈┈┈┈┈┈┈
• Make domain id abstract and provide `string_of_id'
• Document that initializing writes are ok using the Field macro with
respect to the memory model. Also highlight that all writes need to
use `caml_modify' (even immediates)
• check that the selectgen 'coeffect' is correct for DLS.get
• More comments needed for domain.c to help the reader:
• around backup thread state machine and where things happen
• domain spawn/join
• comment/check why `install_backup_thread' is called in spawnee and
spawner
• check the reason why domain terminate is using a mutex for join
(rather than a mutex, condvar pair)
Post-5.00
┈┈┈┈┈┈┈┈┈
• Provide a mechanism for the user to retrieve the number of
processors available for use. This can be implemented by libraries
as well.
• add atomic mutable record fields
• add arrays of atomic variables
WG3: Runtime multi-domain safety
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
Multicore OCaml supports systhreads in a backwards compatible
fashion. The execution model remains the same, except transposed to
domains rather than a single execution context.
Each domain will get its own threads chaining: this means that while
only one systhread can execute at a time on a single domain (akin to
trunk), many domains can still execute in parallel, with their
systhreads chaining being independent. To achieve this, a thread table
is employed to allow each domains to maintain their own independent
chaining. Context switching now involves extra care to handle the
backup thread. The backup thread takes care of GC duties when a thread
is currently in a blocking section. Systhreads needs to be careful
about when to signal it.
The tick thread, used to periodically force thread preemption, has
been updated to not rely on signals (as the multicore signaling model
does not allow this to be done efficiently). Instead, we rely on the
interrupt infrastructure of the multicore runtime and trigger an
“external” interrupt, that will call back into systhreads to force a
yield.
The existing Dynlink API was designed decades ago for a web browser
written in OCaml (called "[mmm]") and is stateful. We'll make it
possible to call concurrently in the OCaml 5.0 MVP, but the WG3
decided to start redesigning the Dynlink API to be less stateful.
Code fragments are now stored in a lockfree skiplist to allow multiple
threads to work on the codefrags structures concurrently in a
thread-safe manner. Extra care is required on cleanup (i.e, freeing
unused code fragments entries): this should only happen on one domain,
and this is done at the end of a major cycle. For the interested,
[ocaml-multicore#672] is a recommended read to see the concurrent
skiplist structure now used.
Signals in multicore have the following behaviour, with the WG3
deciding to change their behaviour to allow coalescing multiple
signals from the perspective of the mutator:
• A program with a single domain should have mostly the same signal
behaviour as trunk. This includes the delivery of signals to
systhreads on that domain.
• Programs with multiple domains treat signals in a global fashion. It
is not possible to direct signals to individual domains or threads,
other than the control through thread sigmask. A domain recording a
signal may not be the one executing the OCaml signal handler.
Frame descriptors modifications are now locked behind a mutex to avoid
races if different threads were to try to apply changes to the frame
table at the same time. Freeing up old frame tables is done at the end
of a major cycle (which is a STW section) in order to be sure that no
thread will be using this old frame table anymore.
Multicore OCaml contains a version of eventlog that is safe for
multiple domains. It achieves this by having a separate CTF file per
domain but this is an interim solution. We hope to replace this
implementation with an existing prototype based on per-domain ring
buffers which can be consumed programmatically from both OCaml and
C. This will be a generalisation of eventlog, and so we should be able
to remove the existing interface if it's not widely adopted yet.
[mmm] <https://caml.inria.fr/pub/old_caml_site/~rouaix/mmm/>
[ocaml-multicore#672]
<https://github.com/ocaml-multicore/ocaml-multicore/pull/672>
Pre-MVP
┈┈┈┈┈┈┈
• Rewrite intern.c so that it doesn't do GC. This code is performance
sensitive as the compiler reads the cmi files by unmarshaling them.
• Benchmark on `big.ml' (from @stedolan) and binary tree benchmark
(from @xavierleroy).
• Ensure the `m->waiters' atomics in systhreads are correct and safe.
• Write down options for `Thread.exit' to be discussed during or after
merge, and what to do if just one domain exits while others continue
to run. Should not be a blocking issue. Changing semantics is ok
from vanilla trunk.
• `m->busy' is not atomic anymore as of
[ocaml-multicore/ocaml-multicore#740], should be reviewed and
merged.
• Restrict `Dynlink' to domain 0 as it is a mutable interface and
difficult to use concurrently.
• Signals stack should move from counting to coalescing semantics.
• Try to delay signal processing at domain spawn so that `Caml_state'
is valid.
• Remove `total_signals_pending' if possible.
[ocaml-multicore/ocaml-multicore#740]
<https://github.com/ocaml-multicore/ocaml-multicore/pull/740>
Post-MVP for 5.00
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
• Probe opam for eventlog usage (introduced in OCaml 4.13) to
determine if removing it will break any applications.
• Eventring merge is OK, eventlog API can be changed if functionality
remains equivalent.
• (could be post 5.00 as well) TLS for systhreads.
Post-5.00
┈┈┈┈┈┈┈┈┈
• Get more data on Dynlink usage and design a new API that is less
stateful.
• @xavierleroy suggested redesigning marshalling in light of the new
allocator.
WG4: Stdlib changes
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
The main guiding principle in porting the Stdlib to OCaml 5.00 is that
1. OCaml 5.00 does not provide thread-safety by default for mutable
data structures and interfaces.
2. OCaml 5.00 does ensure memory-safety (no crashes) even when stdlib
is used in parallel by multiple domains.
3. Observationally pure interfaces remain so in OCaml 5.00.
For OCaml libraries with specific mutable interfaces (e.g. Queue,
Hashtbl, Stack, etc.) they will not be made domain-safe to avoid
impacting sequential performance. Programs using parallelism will need
to add their own lock safety around concurrent access to such
modules. Modules with top-level mutable state (e.g. Filename, Random,
Format, etc..) will be made domain-safe. Some, such as Random, are
being extensively redesigned to use new approaches such as splittable
prngs. The motivation for these choices and further discussion is
found in the [Multicore OCaml wiki page].
The WG4 also noted that we would accept alternative versions of
mutable stdlib modules that are concurrent-safe (e.g. have a
`Concurrent.Hashtbl'), and also hopes to see more lockfree libraries
developed independently by the OCaml community. Overall, WG4
recognised the importance of community involvement with the process of
porting OCaml libraries to parallel safety. We aim to add ocamldoc
tags to the language to mark modules/functions safety, and hope to get
this in the new unified package db at [v3.ocaml.org] ahead of OCaml
5.0.
[Multicore OCaml wiki page]
<https://github.com/ocaml-multicore/ocaml-multicore/wiki/Safety-of-Stdlib-under-Multicore-OCaml>
[v3.ocaml.org] <https://v3.ocaml.org/packages>
◊ Lazy
Lazy values in OCaml allow deferred computations to be run by
*forcing* them. Once the lazy computation runs to completion, the lazy
is updated such that further forcing fetches the result from the
previous forcing. The minor GC also short-circuits forced lazy values
avoiding a hop through the lazy object. The implementation of lazy
uses [unsafe operations from the Obj module].
The implementation of Lazy has been made thread-safe in OCaml
5.00. For single-threaded use, the Lazy module preserves backwards
compatibility. For multi-threaded use, the Lazy module adds
synchronization such that on concurrent forcing of an unforced lazy
value from multiple domains, one of the domains will get to run the
deferred computation while the other will get a new exception
`RacyLazy' .
[unsafe operations from the Obj module]
<https://github.com/ocaml/ocaml/blob/trunk/stdlib/camlinternalLazy.ml>
◊ Random
With [ocaml-multicore#582], we have domain-local PRNGs following
closely along the lines of stock OCaml. In particular, the behaviour
remains the same for sequential OCaml programs. But the situation for
parallel programs is not ideal. Without explicit initialisation, all
the domains will draw the same initial sequence.
There is ongoing discussion on splittable PRNGs in [ocaml/RFCs#28],
and a re-implementation of Random using the Xoshiro256++ PRNG in
[ocaml/ocaml#10701].
[ocaml-multicore#582]
<https://github.com/ocaml-multicore/ocaml-multicore/pull/582>
[ocaml/RFCs#28] <https://github.com/ocaml/RFCs/pull/28>
[ocaml/ocaml#10701] <https://github.com/ocaml/ocaml/pull/10701>
◊ Format
The Format module has some hidden global state for implementing
pretty-printing boxes. While the module has explicit API for passing
the formatter state to the functions, there are predefined formatters
for `stdout' , `stderr' and standard buffer, whose state is maintained
by the module.
The Format module has been made thread-safe for predefined
formatters. We use domain-local versions of formatter state for each
domain, lazily switching to this version when the first-domain is
spawned. This preserves the performance of single-threaded code, while
being thread-safe for multi-threaded use case. See the discussion in
[ocaml/ocaml#10453] for a summary.
[ocaml/ocaml#10453]
<https://github.com/ocaml/ocaml/issues/10453#issuecomment-868940501>
◊ Mutex, Condition, Semaphore
The Mutex, Condition and Semaphore modules are the same as systhreads
in stock OCaml. They now reside in `stdlib' . When systhreads are
linked, the same modules are used for synchronization between
systhreads.
Pre-MVP
┈┈┈┈┈┈┈
• Mark lazy as not thread safe.
• Unify RacyLazy and Undefined
• Remove domain-local unique token
• Remove try_force
• Add the Bucket module for ephemerons with a default sequential
implementation as seen in OCaml 4.13.
Post-MVP for 5.00
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
• Introduce ocamldoc tags for different concurrency safety
• domain-safe
• systhread-safe
• fiber-safe
• not-concurrency-safe (= !domain-safe || !systhread-safe ||
!fiber-safe) – also used as a placeholder for libraries and
functions not analysed for concurrency.
• Add documentation for memory model in the manual. Specifically, no
values out of thin air – no need to precisely document the memory
model aside from pointing to paper.
• For `Arg' module, deprecate current but not whole module
• remove ThreadUnix as a simple module that would no longer need Unix.
• Dynlink should have a mutex inside it to ensure it doesnt crash
especially in bytecode.
Post-5.00
┈┈┈┈┈┈┈┈┈
• Atomic arrays
• Ephemerons reimplemented in terms of Bucket module.
• Make disjoint the update of the lazy tag and marking by using
byte-sized write and CAS.
WG5: Fibers
┄┄┄┄┄┄┄┄┄┄┄
Fibers are the runtime system mechanism that supports effect
handlers. The design of effect handlers in OCaml has been written up
in the [PLDI 2021 paper].The motivation for adding effect handlers and
some more examples are found in [these slides].
[PLDI 2021 paper] <https://arxiv.org/abs/2104.00250>
[these slides]
<https://speakerdeck.com/kayceesrk/effect-handlers-in-multicore-ocaml>
◊ Programming with effect handlers
Effect handlers are made available to the OCaml programmer from
`stdlib/effectHandlers.ml' (although this will likely be renamed
`Effect' soon). The EffectHandlers module exposes two variants of
effect handlers – deep and shallow. Deep handlers are like folds over
the computation tree whereas shallow handlers are akin to [case
splits]. With deep handlers, the handler wraps around the
continuation, whereas in shallow handlers it doesn’t.
Here is an example of a program that uses deep handlers to model
something analogous to the `Reader' monad.
┌────
│ open EffectHandlers
│ open EffectHandlers.Deep
│
│ type _ eff += Ask : int eff
│
│ let main () =
│ try_with (fun _ -> perform Ask + perform Ask) ()
│ { effc = fun (type a) (e : a eff) ->
│ match e with
│ | Ask -> Some (fun (k : (a,_) continuation) -> continue k 1)
│ | _ -> None }
│
│ let _ = assert (main () = 2)
└────
Observe that when we resume the continuation `k' , the subsequent
effects performed by the computation are also handled by the same
handler. As opposed to this, for the shallow handler doesn’t. For
shallow handlers, we use `continue_with' instead of continue.
┌────
│ open EffectHandlers
│ open EffectHandlers.Shallow
│
│ type _ eff += Ask : int eff
│
│ let main () =
│ let rec loop (k: (int,_) continuation) (state : int) =
│ continue_with k state
│ { retc = (fun v -> v);
│ exnc = (fun e -> raise e);
│ effc = fun (type a) (e : a eff) ->
│ match e with
│ | Ask -> Some (fun (k : (a, _) continuation) -> loop k 1)
│ | _ -> None }
│ in
│ let k = fiber (fun _ -> perform Ask + perform Ask) in
│ loop k 1
│
│ let _ = assert (main () = 2)
└────
Observe that with a shallow handler, the recursion is
explicit. Shallow handlers makes it easier to encode cases where state
needs to be threaded through. For example, here is a variant of the
`State' handler that encodes a counter:
┌────
│ open EffectHandlers
│ open EffectHandlers.Shallow
│
│ type _ eff += Next : int eff
│
│ let main () =
│ let rec loop (k: (int,_) continuation) (state : int) =
│ continue_with k state
│ { retc = (fun v -> v);
│ exnc = (fun e -> raise e);
│ effc = fun (type a) (e : a eff) ->
│ match e with
│ | Next -> Some (fun (k : (a, _) continuation) -> loop k (state + 1))
│ | _ -> None }
│ in
│ let k = fiber (fun _ -> perform Next + perform Next) in
│ loop k 0
│
│ let _ = assert (main () = 3)
└────
While this encoding is possible with deep handlers (by the usual
`State' monad trick of building up a computation using a closure), it
feels more natural with shallow handlers. In general, one can easily
encode deep handlers using shallow handlers, but going the other way
is challenging. With the typed effects work currently in development,
the default would be shallow handlers and deep handlers would be
encoded using the shallow handlers.
As a bit of history, the current implementation is tuned for deep
handlers and has gathered optimizations over several iterations. If
shallow handlers becomes more widely in the coming years, it may be
possible to put in some tweaks that removes a few allocations. That
said, the semantics of the deep and shallow handlers in this future
implementation will remain the same as what is currently in OCaml 5.00
branch.
[case splits]
<https://www.dhil.net/research/papers/generalised_continuations-jfp-draft.pdf>
Post-MVP for 5.00
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
• Add ARM64 backend
• Documentation on the usage of effect handlers.
• Current stack size should be the sum of the stack sizes of the stack
of fibers. Currently, it only captures the top fiber size.
⁃ This is not straight-forward as it seems. Resuming continuations
attaches a stack. Should we do stack overflow checks there? I'd
not, as this would make resuming continuations slower. One idea
might be to only do the stack overflow check at stack realloc,
which catches the common case.
Post-5.00
┈┈┈┈┈┈┈┈┈
• Add support for compiling with frame pointers.
The November activities
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
That wraps up the mammoth code review summary, and significant
decisions taken. Overall, we are full steam ahead for generating an
OCaml 5.0 PR, although we do have our work cut out for us in the
coming months! Now we continue with our regular report on what else
happened in November.The ecosystem is continuing to evolve, and there
are significant updates to Eio, the Effects-based parallel IO for
OCaml.
[Lwt.5.5.0] has been released that supports dispatching pure
computations to multicore domains. The Sandmark benchmarking has now
been updated to build for 5.00, and the current-bench tooling is being
improved to better track the performance analysis and upstream merge
changes.
As always, the Multicore OCaml updates are listed first, which contain
the upstream efforts, documentation changes, and PR fixes. This is
followed by the ecosystem updates to `Eio' and `Tezos'. The Sandmark,
sandmark-nightly and current-bench tasks are finally listed for your
kind reference.
/The full release notes can be found at the [archive link] above./
[Lwt.5.5.0]
<https://discuss.ocaml.org/t/ann-lwt-5-5-0-lwt-domain-0-1-0-lwt-react-1-1-5/8897>
[archive link]
<https://discuss.ocaml.org/t/multicore-ocaml-november-2021-with-results-of-code-review/8934/1>
New release of Menhir (20211128)
════════════════════════════════
Archive:
<https://sympa.inria.fr/sympa/arc/caml-list/2021-12/msg00000.html>
François Pottier announced
──────────────────────────
The recent release of Menhir (20211125) creates some difficulties with
OCaml versions 4.07 to 4.10, where it triggers a type-checker bug
(fixed in 4.11).
A new release of Menhir (20211128) appears today and is expected to
eliminate these problems.
┌────
│ opam update
│ opam install menhir.20211128
└────
Advent of Code 2021
═══════════════════
Archive: <https://discuss.ocaml.org/t/advent-of-code-2021/8945/1>
Shon announced
──────────────
For any other OCamlers having a go at [Advent of Code] this year, I
thought I'd create this thread for discussing/sharing/comparing.
I doubt I'll see it all the way through myself, but will keep at it
until I run out of steam :)
If anyone would like to form an ocamlcentric leaderboard, all may feel
free to use my private leaderboard, by entering code `221063-e41acad3'
at <https://adventofcode.com/2021/leaderboard/private> (after logging
in). I'd also be happy to join a different leaderboard if there's
already one fit for this purpose :)
(Discussion from a few years back is here: here:
<https://discuss.ocaml.org/t/advent-of-code-starts-today/1223>)
[Advent of Code] <https://adventofcode.com/>
"What is an Operating System?": Anil Madhavapeddy on Signals and Threads
════════════════════════════════════════════════════════════════════════
Archive:
<https://discuss.ocaml.org/t/what-is-an-operating-system-anil-madhavapeddy-on-signals-and-threads/8949/1>
Shon announced
──────────────
<https://signalsandthreads.com/what-is-an-operating-system/>
Other OCaml News
════════════════
From the ocamlcore planet blog
──────────────────────────────
Here are links from many OCaml blogs aggregated at [OCaml Planet].
• [Release of Frama-C 24.0 (Chromium)]
• ['Signals and Threads' Podcast: What is an Operating System?]
• [The proposal for a proof assistants StackExchange site]
• [Tarides and Hyper: Partners in Agricultural Innovation]
• [opam releases: 2.0.10, 2.1.1, and opam depext 1.2!]
• [MirageOS Workshop: Working with the Raspberry Pi 4]
• [MirageOS 4.0 Preview Live Presentation]
• [Isolating Xwayland in a VM]
[OCaml Planet] <http://ocaml.org/community/planet/>
[Release of Frama-C 24.0 (Chromium)]
<https://frama-c.com/fc-versions/chromium.html>
['Signals and Threads' Podcast: What is an Operating System?]
<https://tarides.com/blog/2021-11-23--signals-and-threads-podcast-what-is-an-operating-system>
[The proposal for a proof assistants StackExchange site]
<http://math.andrej.com/2021/11/20/proof-assistants-stackexchange-site/>
[Tarides and Hyper: Partners in Agricultural Innovation]
<https://tarides.com/blog/2021-11-18-tarides-hyper-partners-in-agricultural-innovation>
[opam releases: 2.0.10, 2.1.1, and opam depext 1.2!]
<http://opam.ocaml.org/blog/blog/opam-2-0-10-2-1-1-depext/>
[MirageOS Workshop: Working with the Raspberry Pi 4]
<https://tarides.com/blog/2021-11-11-mirageos-workshop-working-with-the-raspberry-pi-4>
[MirageOS 4.0 Preview Live Presentation]
<https://tarides.com/blog/2021-11-09-mirageos-4-0-preview-live-presentation>
[Isolating Xwayland in a VM]
<https://roscidus.com/blog/blog/2021/10/30/xwayland/>
Old CWN
═══════
If you happen to miss a CWN, you can [send me a message] and I'll mail
it to you, or go take a look at [the archive] or the [RSS feed of the
archives].
If you also wish to receive it every week by mail, you may subscribe
[online].
[Alan Schmitt]
[send me a message] <mailto:alan.schmitt at polytechnique.org>
[the archive] <https://alan.petitepomme.net/cwn/>
[RSS feed of the archives] <https://alan.petitepomme.net/cwn/cwn.rss>
[online] <http://lists.idyll.org/listinfo/caml-news-weekly/>
[Alan Schmitt] <https://alan.petitepomme.net/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/caml-news-weekly/attachments/20211207/95331da1/attachment-0001.html>
More information about the caml-news-weekly
mailing list