a event-loop.=20
You create Deferred.t and it might be added to a queue and a scheduler
behind might be adjusting the order of running for all Deferred.t in
the queue.=20
Am I correct?
Q2: Deferred.return and Deferred.bind
If I say
Deferred.return 1
It will returns me a Deferred.t, but inside the function return or
bind somehow an "event" is implicitly added to the default queue for
scheduling, right?
If I am correct above,=20
Q3: Is Async depending on -thread? The queue or scheduler needs
compiler support?=20
I just need to understand the whole picture in a rough way first.
=20=20=20=20=20=20
** David House then replied:
There is a queue of jobs in the scheduler. The scheduler runs the jobs
one by one. Jobs may schedule other jobs. A job is a pair of
['a * 'a -> unit].
There's a thing called a deferred. ['a Deferred.t] is an initially
empty box that may become filled later with something of type ['a].
There is a similar type called ['a Ivar.t] -- the difference is that
ivars have a function to actually fill in the value, whereas deferreds
do not: a deferred is a "read-only" view on an ivar.
You can wait on a deferred using bind. Doing [x >>=3D f] mutates the
deferred x to add f as a "handler". When a deferred is filled, it adds
a job to the scheduler for each handler it has.=20
Doing [Deferred.return 1] allocates a deferred which is already filled
and has no handlers. Binding on that will immediately schedule a job
to run your function. (The job is still scheduled though, rather than
being run immediately, to ensure that you don't have an immediate
context switch -- in async, the only context switch points are the
binds.)
The primitive operations that block are replaced with functions that
return deferreds, and go do their work in a separate thread. There's a
thread pool to make sure you don't use infinity threads. (I think the
default cap is 50 threads.) I think yes, async does depend on -thread.
There is an important optimisation: if you want to read or write to
certain file descriptors, that doesn't use a thread. Instead there's a
central list of such file descriptors. There's also a central list of
all "timer events" (e.g. deferreds that become deferred after some
amount of time). The scheduler actually is based around a select loop:
it does the following:
run all the jobs
if more jobs have been scheduled, run those too
keep going until there are no more jobs, or we hit the
maximum-jobs-per-cycle cap
sleep using select until one read fd is read, or a write fd is ready,
or a timer event is due to fire
do that thing
There's also a way to manually interrupt the scheduler. Blocking
operations other than reading/writing to fds do this: they run in a
thread, grab the async scheduler lock, fill in an ivar, then wake up
the scheduler to ensure timely running of the jobs they just
scheduled. The async scheduler lock is necessary because the scheduler
itself is not re-entrant: you cannot have multiple threads modifying
the scheduler's internals.
=20=20=20=20=20=20
** Dan Stark then asked and Ashish Agarwal replied:
> Thank you very much for this comprehensive explanation.
>=20
> Can I also know who is responsible for the queue and scheduler?=20
>=20
> Are they created and maintained by OCaml thread (OCaml internal) or
> Async (3rd party library, which means Async create the job queue and
> has its own scheduler)?=20
>=20
> In addition, will the compiler got involved in handling Deferred.t?
>=20
> I ask above questions because I felt quite curious about what is
> happening in the followings:
>=20
> Suppose we have a normal function:
>=20
> let f1 () =3D print_endline "hello"; whatever_result;;
>=20
> Normally, no matter what whatever_result is, when I do let _ =3D f1
> ();;, print_endline "hello" will be executed, am I right? For example,
> finally returning an int or a record or a lazy.t, etc, "hello" would
> be printed out.
>=20
> However, if I do
>=20
> let f2 () =3D print_endline "hello"; return 1;;
>=20
> let _ =3D f2 ();; would do nothing unless I run the schedule let _ =3D
> ignore(Scheduler.go());;=20
>=20
> Since for f2 I am not using any other special creation function and
> the only special bit is return 1 after print_endline, if the compiler
> doesn't get involved, how can compiler know the whole application of
> f2() should be in future execution?=20
When you use Async, you must do `open Async.Std`, which overrides all
blocking functions from the standard library. Thus, in f2, it's not
that the "return 1" part somehow changes the behavior of the previous
code. Rather, since you've written "return 1", you've presumably done
`open Async.Std`, so the print_endline function is actually the one
from Async. So no, the compiler doesn't get involved. Async is
implemented purely as a library.
=20=20=20=20=20=20
** Yaron Minsky also added:
For what it's worth, this gives a decent overview of Async (if I do
say so myself)
<https://realworldocaml.org/v1/en/html/concurrent-programming-with-async.ht=
ml>
This isn't quite right, but a good mental model is to think of Async
as being single threaded. Scheduler.go starts up the async scheduler
on the main thread, and you do indeed need to do that for any IO to
actually happen.
=20=20=20=20=20=20
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
3) Multicore runtime
Archive: <https://sympa.inria.fr/sympa/arc/caml-list/2014-06/msg00029.html>
------------------------------------------------------------------------
** Jiten Pathy asked and Stephen Dolan replied:
> Is there any public available code corresponding [1] with some
> progress, if any? Thank you.
>
> 1. <https://github.com/ocamllabs/compiler-hacking/wiki/Multicore-runtime>
There is, but "pre-alpha" does not begin to describe it.
<https://github.com/stedolan/ocaml>
The branch is frequently rebased, in order to present a sort-of
readable patch series instead of accurately describing the convoluted
development history. So, if you clone this, then a) it might not build
without some makefile hacking, and b) git pull will break after the
next rebase.
Currently, there's a mostly-working parallel GC, but many features are
broken (such as ocamlopt, weak pointers, finalisers, and a few
others). Also, the runtime does not yet export enough primitives to
OCaml code to do anything interesting, so it's not useful for more
than GC development at the moment. Still, progressing nicely.
So, feel free to look around, but don't get your hopes up too soon :)
=20=20=20=20=20=20
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
4) PG'OCaml 2.1
Archive: <https://sympa.inria.fr/sympa/arc/caml-list/2014-06/msg00031.html>
------------------------------------------------------------------------
** Dario Teixeira announced:
No earth-shattering news for this release. Version 2.0 (released last
week) relied on the List.iteri and List.mapi functions available since
OCaml 4.00.0. However, it turns out we have users still on OCaml
3.12.1 (or maybe older). This release simply adds those missing
functions for compatibility with older OCaml releases.
For more information, please visit the project's homepage:
<http://pgocaml.forge.ocamlcore.org/>
Also, the new package should be hitting OPAM soon.
=20=20=20=20=20=20
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
5) Thoughts on targeting windows
Archive: <https://sympa.inria.fr/sympa/arc/caml-list/2014-06/msg00039.html>
------------------------------------------------------------------------
** William asked:
we are considering using OCaml for a rather large project, the bulk of
which will be networking and encryption. OCaml seems to meet our needs
with one exception:=20
we'd like to target windows (as well as linux & mac) and we got the
impression that this would be complicated -- we gathered that neither
jane street's Core nor OPAM are windows compatible.=20
Would still recommend using OCaml? Are there workarounds, or other
libraries that would replace Core?
=20=20=20=20=20=20
** Virgile Prevosto suggested:
regarding OCaml package management under Windows, you should have a
look at wodi:
<http://wodi.forge.ocamlcore.org/>
The package page lists core_kernel, batteries and extlib, but as I use
none of those I can't tell how well they work on Windows.
=20=20=20=20=20=20
** John Whitington also suggested:
Here is the Windows installer:
<http://protz.github.io/ocaml-installer/>
This also installs 'ocamlfind', which means that you can download
source packages for libraries you're interested in, compile them,
install them and use them relatively easily. Not as convenient as
OPAM, of course.
Here is a library for cryptography:
<https://forge.ocamlcore.org/projects/cryptokit/>
Here is a library for concurrency which runs on Windows:
<http://ocsigen.org/lwt/>
Here is a different Standard Library replacement/augmentation:
<https://github.com/ocaml-batteries-team/batteries-included>
(The software I build for Windows, whilst complex in terms of what it
does, is just a single statically linked executable which reads
a file, processes it and writes a file, so I can't tell you anything
about networking under Windows.)
=20=20=20=20=20=20
** David Allsopp also replied:
I believe Core_kernel aims to be the platform-neutral parts of core?
There are other Jane Street libs which compile just fine on
Windows. Batteries, as others have noted, works out of the
box. Usually, I find that the biggest problem in third party libs is
in build systems (becoming less so with Oasis, OCamlbuild and so on)
making na=C3=AFve decisions about Windows but that doesn't usually take
much patching.
Most of what I do is Windows-oriented, but some of what I've done is
Windows and Linux. My experience is that it's important to keep
Windows in the picture early on to avoid pain later - so ensure that
daily builds are working on Windows or perhaps that one of your
developers is always working on Windows or something... that should
avoid accidentally selecting a Unix-only library and only realising
that a painfully long way down the road (or that the library you
thought was cross-platform contains an assert false for the function
you need when running on Windows!). If you write something which works
on Windows in OCaml it will probably translate with little pain to
Linux but the reverse isn't necessarily true.
While OPAM is great, I personally find that downloading and compiling
a library, even by hand, represents an insignificant amount of time
compared with reading its documentation, evaluating its samples and so
on in the overall process of working out whether I want to use
a component... but apparently the pain of not having a package manager
really, really, really hurts people coming from the Unix world ;o)
=20=20=20=20=20=20
** Yaron Minsky, Sebastien Mondet, and Thomas Gazagnaire then had this exch=
ange:
(Yaron Minsky)
>>> Core_kernel is pure OCaml, and so should work fine on Windows (and Java=
script!)
(Sebastien Mondet)
>> Actually some dependencies of core_kernel have C code, like bin_prot:
>> <https://github.com/janestreet/bin_prot/blob/master/lib/blit_stubs.c>
(Yaron Minsky)
> Indeed! I forgot about that last little bit. We killed most of the C,
> but there's a bit left. That said, those should be pretty portable,
> and can (and have been) stubbed out so it can work in a Javascript
> context.
(Thomas Gazagnaire)
Indeed, little changes are needed to compile core_kernel to
javascript. See <https://github.com/janestreet/core_kernel/pull/8>
=20=20=20=20=20=20
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
6) uCorelib 0.2.0
Archive: <https://sympa.inria.fr/sympa/arc/caml-list/2014-06/msg00044.html>
------------------------------------------------------------------------
** Yoriyuki Yamagata announced:
I'm pleased to announce uCorelib 0.2.0, a new and simple Unicode
library for OCaml.
<https://github.com/yoriyuki/ucorelib/releases/tag/v0.2.0>
This release add=20
- Persistent set of Unicode characters. USet
- Persistent map from Unicode characters to arbitrary types. UMap
- Readonly, fast access, compact map from Unicode to any type, bool,
and char. UCharTbl
=20=20=20=20=20=20
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
7) Other OCaml News
------------------------------------------------------------------------
** From the ocamlcore planet blog:
Thanks to Alp Mestan, we now include in the OCaml Weekly News the links to =
the
recent posts from the ocamlcore planet blog at <http://planet.ocaml.org/>.
PG'OCaml 2.1 released:
<https://forge.ocamlcore.org/forum/forum.php?forum_id=3D902>
OCaml bindings to Tk:
<https://forge.ocamlcore.org/projects/otk/>
Python to OCaml: retrospective:
<http://roscidus.com/blog/blog/2014/06/06/python-to-ocaml-retrospective/>
The Minnesota Goodbye:
<http://www.somerandomidiot.com/blog/2014/06/03/the-minnesota-goodbye/>
Sr. Software Engineer at Clutch Analytics (Full-time):
<http://functionaljobs.com/jobs/8720-sr-software-engineer-at-clutch-analy=
tics>
=20=20=20=20=20=20
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Old cwn
------------------------------------------------------------------------
If you happen to miss a CWN, you can send me a message
(alan.schmitt at polytechnique.org) and I'll mail it to you, or go take a look=
at
the archive (<http://alan.petitepomme.net/cwn/>) or the RSS feed of the
archives (<http://alan.petitepomme.net/cwn/cwn.rss>). If you also wish
to receive it every week by mail, you may subscribe online at
<http://lists.idyll.org/listinfo/caml-news-weekly/> .
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D