[cwn] Attn: Development Editor, Latest OCaml Weekly News

[Alan Schmitt]

Hello

Here is the latest OCaml Weekly News, for the week of April 02 to 
09,
2019.

Table of Contents
─────────────────

Easy_logging 0.2
OCaml Users and Developers Meeting 2019
Firewall-tree - demo using MirageOS is available, with overview of 
current progress
routes: path based routing for web applications
Other OCaml News
Old CWN


Easy_logging 0.2
════════════════

  Archive: <https://discuss.ocaml.org/t/easy-logging-0-2/3551/6>


Sapristi announced
──────────────────

  Version 0.4 is out (or should be soon) :

  • printf style logging is now the default
  • simplifed configuration in case of multiple loggers (and 
  closer to
    the python module) :
    • the loggers form a tree (based on their name, dots 
    indicating
      descendence)
    • log items are passed to the handlers of a logger's ancestors 
    (so
      that few handlers need initialisation) (possible to 
      override)
    • loggers inherit the level of their ancestor if not set 
    explicitely
  • an additional package `easy_logging_yojson' provides 
  initialisation
    of loggers from the json format (with `ppx_deriving_yojson'), 
    so
    that configuration can be fully done outside the code.


OCaml Users and Developers Meeting 2019
═══════════════════════════════════════

  Archive:
  <https://discuss.ocaml.org/t/ocaml-users-and-developers-meeting-2019/3620/1>


David Allsopp announced
───────────────────────

  I'm pleased to annouce that the 8th ICFP co-located OCaml Users 
  and
  Developers Workshop will be held in Berlin on August 23rd.

  The workshop is an informal occasion consisting of ~20 minute 
  talks
  and a poster session on topics ranging from compiler 
  developments, new
  libraries and applications through to experience reports on
  industrial/experimental uses of OCaml.

  The [submission site] for talk proposals is now open and full 
  details
  of the call may be found on the [ICFP 2019 website]. The 
  deadline for
  proposals is May 17 [Anywhere on Earth].


[submission site] <https://ocaml2019.hotcrp.com>

[ICFP 2019 website]
<https://icfp19.sigplan.org/home/ocaml-2019#Call-for-Presentations>

[Anywhere on Earth] 
<https://en.wikipedia.org/wiki/Anywhere_on_Earth>


Firewall-tree - demo using MirageOS is available, with overview of 
current progress
═══════════════════════════════════════════════════════════════════════════════════

  Archive:
  <https://discuss.ocaml.org/t/firewall-tree-demo-using-mirageos-is-available-with-overview-of-current-progress/3621/1>


Darren announced
────────────────

  I'm very pleased to announce that [firewall-tree] library has 
  reached
  a presentable stage and a demo which uses firewall-tree in 
  MirageOS is
  available.


[firewall-tree] <https://gitlab.com/darrenldl/ocaml-firewall-tree>

Demo
╌╌╌╌

  The demo code is available [here].

  The code itself is quite thoroughly commented, the README in the
  folder contains build and run instructions and also 
  illustrations of
  the tree in graphical form, so I won't dive into the details of 
  the
  demo here.

  In short, the demo shows how to separate of ICMP trafffic and 
  TCP
  traffic. For TCP, filter some TCP traffic, then load balance the
  remaining TCP connections across multiple destination IPv4
  addresses. For ICMP ping requests, only reply to every other 
  one.

  The tree itself is only a couple of lines, but there is quite a 
  bit
  setup code for wrapping MirageOS stuff into a module and types 
  usable
  by firewall-tree. However, it should be possible to package the 
  setup
  code into a library supporting MirageOS as a firewall-tree 
  backend, so
  in future the usage of the library should be much smoother and 
  less
  cluttered than the demo.

  I'd like to note that the library right now is still very rough 
  at
  more complex constructs, and the connection tracker, which is 
  used
  inside the translators, works fine for a very typical TCP 
  connection,
  but fails to handle TCP retransmissions, or TCP PDUs with RST 
  and PSH
  flags.


[here]
<https://github.com/darrenldl/firewall-tree-demos/tree/master/demos/load-balancing-proxy>


Library overview
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  The library requires a "base", which you can think of as an
  environment implementation (see [here] for module type 
  signature),
  which it then extends into a full firewall module. The resulting
  module provides the following functions/constructs
  • [Hierarchical PDU definition]
  • [PDU decapsulation]
  • [PDU mapping] (can also map inner components)
  • [PDU replacement] (can also replace inner components)
  • [PDU pretty printing] (see the demo for sample output)
  • [Memory bounded lookup table] with evict timed out entries 
  function
  • [Routing logic unit] only partial implementaiton is in place
    currently
  • [Connection tracker] (bidirectional, so you can use it without 
  much
    thinking)
    • Largely functional, still a long way to go before becoming 
    robust
      enough
  • [Predicates for building filters/selectors]
  • [Translators] for building load balancers or proxy
    • These were essentially a more general form of NAT 
    translators when
      I wrote them, so hopefully they (or variant of them) can be 
      used
      to build a full blown NAT at some point

  The package provides another set of functors which take the 
  resulted
  tree module as parameter and derive commonly used constructs. 
  See demo
  for example use.
  • [Selectors]
  • [Modifiers]
  • [Scanners]


[here]
<https://darrenldl.gitlab.io/ocaml-firewall-tree/doc/firewall-tree/Firewall_tree/module-type-B/index.html>

[Hierarchical PDU definition]
<https://darrenldl.gitlab.io/ocaml-firewall-tree/doc/firewall-tree/Firewall_tree/module-type-S/PDU/index.html>

[PDU decapsulation]
<https://darrenldl.gitlab.io/ocaml-firewall-tree/doc/firewall-tree/Firewall_tree/module-type-S/PDU_to/index.html>

[PDU mapping]
<https://darrenldl.gitlab.io/ocaml-firewall-tree/doc/firewall-tree/Firewall_tree/module-type-S/PDU_map/index.html>

[PDU replacement]
<https://darrenldl.gitlab.io/ocaml-firewall-tree/doc/firewall-tree/Firewall_tree/module-type-S/PDU_replace/index.html>

[PDU pretty printing]
<https://darrenldl.gitlab.io/ocaml-firewall-tree/doc/firewall-tree/Firewall_tree/module-type-S/To_debug_string/index.html>

[Memory bounded lookup table]
<https://darrenldl.gitlab.io/ocaml-firewall-tree/doc/firewall-tree/Firewall_tree/module-type-S/Lookup_table/index.html>

[Routing logic unit]
<https://darrenldl.gitlab.io/ocaml-firewall-tree/doc/firewall-tree/Firewall_tree/module-type-S/RLU_IPv4/index.html>

[Connection tracker]
<https://darrenldl.gitlab.io/ocaml-firewall-tree/doc/firewall-tree/Firewall_tree/module-type-S/Conn_track/index.html>

[Predicates for building filters/selectors]
<https://darrenldl.gitlab.io/ocaml-firewall-tree/doc/firewall-tree/Firewall_tree/module-type-S/Pred/index.html>

[Translators]
<https://darrenldl.gitlab.io/ocaml-firewall-tree/doc/firewall-tree/Firewall_tree/module-type-S/Translator/index.html>

[Selectors]
<https://darrenldl.gitlab.io/ocaml-firewall-tree/doc/firewall-tree/Firewall_tree/Selectors/module-type-S/index.html>

[Modifiers]
<https://darrenldl.gitlab.io/ocaml-firewall-tree/doc/firewall-tree/Firewall_tree/Modifiers/module-type-S/index.html>

[Scanners]
<https://darrenldl.gitlab.io/ocaml-firewall-tree/doc/firewall-tree/Firewall_tree/Scanners/module-type-S/index.html>


Moving onward
╌╌╌╌╌╌╌╌╌╌╌╌╌

  Still a lot of things to be done, following lists a few of the 
  items
  • TCP connection tracking needs to be fixed
  • Implement more translators, make them more modular (e.g. 
  swappable
    algorithms)
  • Couple translators to form NAT modules
  • Benchmark
    • I am hoping this library doesn't slow down MirageOS too much 
    for
      equivalent functionalities, but I don't have concrete 
      benchmarks


Final thoughts
╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  The progress so far has been quite time consuming to implement, 
  with a
  lot of redesigns and rewrites in-between. I am hoping the 
  current
  design is sane, but I feel I've invested too much time in last 
  couple
  of weeks to get a clear view of things. So I would absolutely
  appreciate any feedback, whether it's on API design, the 
  architecture
  of the code or any other aspects.

  I had some experiences with networking and firewall, but as far 
  as
  implementing a network stack goes, I have zero experience, so 
  I'd also
  appreciate help on components which demand such expertise as 
  well.


Hannes Mehnert asked and Darren replied
───────────────────────────────────────

  Thanks for reading through!

  > I’ve not been reading through all of your code, but you seem 
  > to
    [have implemented 80% of a TCP stack] and also mention that 
    reset
    handling (what about simultaneous open, (IP) fragmentation, 
    and
    retransmissions?) is missing.

  • Simultaneous open
    • Uh…uh oh, I didn't realise simultaneous open is a thing
    • This might require redesign of the TCP connection tracker as 
    it
      right now assumes only one side is the initiator, and the 
      other
      side must be the responder
  • IP fragmentation
    • Don't think this matters too much unless the library moves 
    toward
      supporting deep packet inspection?
    • But yeah this may be a problem if the library needs to 
    support
      moving packets from a layer 2 segment of higher MTU to one 
      with
      lower MTU? But that should be handled by the IP layer (of
      MirageOS) in this case
    • In any case I don't know if it should be in the scope of the
      library
  • Retransmissions
    • From firewall perspective I think it only needs to remember 
    seeing
      the PDU at some point and just allow duplicates to be sent 
      through
      up to some threshold, which means collecting the hashes 
      should
      suffice I hope

  Ideally the library is network implementation agnostic, but I 
  can't
  tell if that's a reachable goal exactly considering all these
  intricate things.

  > Did you try to use mirage-tcpip instead, which already 
  > implements
    some of the missing pieces (at least reset handling AFAIR)?

  Right now the demo itself uses `Tcp.Tcp_packet' from tcpip to 
  handle
  serialisation and deserialisation, but not flow or any 
  connection
  establishing components.

  I can't tell if flow is suitable for firewall, since that means 
  the
  unikernel needs to add 65535 listeners to cover all ports? I 
  might be
  being silly here or misunderstood something, please correct me 
  if I
  am.

  > But given the [complexity of TCP], I’m not sure how many 
  > partial TCP
    implementations in OCaml are sensible

  W.r.t. stack implementation, I think I might have picked the 
  wrong
  wording. For the library atm, RST handling entails some 
  adjustments in
  the state machine (which is not too much work, but need a lot of
  debugging probably), and retransmission entails recording PDUs
  (through hashes most likely) and allow pass through if 
  duplicates are
  seen (up to N times etc). Both of which are much less work than 
  a full
  stack implementation.

  A firewall does need to work on a slightly different perspective
  compared to a client, however, as when a firewall sees a PDU, 
  it's not
  always the case that either client will see it, as opposed to 
  being a
  client and receiving something means you know for sure you've 
  received
  it. So the other adjustment required is the state machine of
  connection tracker needs to tolerate "partial" state transition 
  in
  some way (so just look ahead one state but don't commit or 
  whatever).

  So overall it's more of a partial stack implementation powerful 
  enough
  to observe and forward things and that's the end of the story.

  Thanks for the link to the paper btw! I'm interested in formal
  verification stuff in general and currently researching on 
  protocol
  verification topics at uni, so I'll definitely give it a 
  thorough read
  later.


[have implemented 80% of a TCP stack]
<https://gitlab.com/darrenldl/ocaml-firewall-tree/blob/master/src/conn_track_ip_generic_tcp.ml>

[complexity of TCP] 
<https://www.cl.cam.ac.uk/~pes20/Netsem/paper3.pdf>


routes: path based routing for web applications
═══════════════════════════════════════════════

  Archive:
  <https://discuss.ocaml.org/t/ann-routes-path-based-routing-for-web-applications/3624/1>


Anurag Soni announced
─────────────────────

  I have just published an initial release for a path based 
  routing
  library. The library has minimal dependencies and isn't tied to 
  any
  particular HTTP or UI framework.

  Example usage can be like below:

  ┌────
  │ module Request = struct
  │   type t
  │   ...
  │ end
  │
  │ module Response = struct
  │   type t
  │   ...
  │ end
  │
  │ let get_user state (id: int) =
  │   (* Request handlers can define their own routes too *)
  │   let open Routes in
  │   let routes = [ int64 </> boolean </> ==> (fun _ i b -> ... ) 
  ] in
  │   match_with_state ~state routes with
  │   | None -> ...
  │   | Some response -> ...
  │
  │ let search_user _ (name: string) (city : string) =
  │   ...
  │
  │ let routes =
  │   let open Routes in
  │   [ empty ==> idx (* matches the index route "/" *)
  │   ; method' `GET </> s "user" </> int </> empty ==> get_user 
  (* matches "/user/<int>" *)
  │   ; method' `GET </> s "user" </> str </> str ==> search_user 
  (* missing empty so it matches "/user/<str>/<str>/*" *)
  │   ]
  │
  │ match Routes.match' routes ~req ~target:"/some/url" ~meth:`GET 
  =
  │ | None -> (* No route matched. Alternative could be to provide 
  default routes *)
  │ | Some r -> (* Match found. Do something further with handler 
  response *)
  └────

  Git repository: <https://github.com/anuragsoni/routes>

  Example using Httpaf:
  <https://github.com/anuragsoni/routes/blob/7fc2ecb0b3a2e46b26254efe54f1147d99d91995/example/main.ml>

  This is not published on `opam' yet so it can be pinned locally 
  via:
  `opam pin add routes 
  git+https://github.com/anuragsoni/routes.git'


Jean Michel asked and Anurag Soni replied
─────────────────────────────────────────

  > Have you looked at [dispatch]?

  > What are the advantages/differences of path?

  Yes i did look at dispatch. One difference from dispatch is that 
  the
  URL's path parameters are extract in a way that assigns them the 
  types
  defined by the user. If such a type coercion isn't possible the 
  route
  won't match.


[dispatch] <https://github.com/inhabitedtype/ocaml-dispatch>


Later on, Anurag Soni added
───────────────────────────

  A small update to the library.

  • Updated the internal representation of a route so the same 
  source is
    used for both scanning and printing routes.
  • Added a `sprintf' like function to format routes.
  • Route matching is now strict by default. ex: `s "user" </> 
  str' will
    just match `/user/<string>' and not `/user/<string>/*'
  • Following from the previous point, nested routing has been 
  removed
    for now.

  ┌────
  │ utop # let route = method' None (s "foo" </> int </> str </> 
  bool);;
  │ - : val route : (int -> string -> bool -> unit -> '_weak1, 
  '_weak1) route =
  │   Route (None, S ("foo", S ("/", Int (S ("/", Str (S ("/", 
  Bool End)))))))
  │
  │ utop # sprintf route;;
  │ - : int -> string -> bool -> unit -> string = <fun>
  │
  │ utop # (sprintf route) 12 "bar" false ();;
  │ - : string = "foo/12/bar/false"
  └────


Other OCaml News
════════════════

From the ocamlcore planet blog
──────────────────────────────

  Here are links from many OCaml blogs aggregated at [OCaml 
  Planet].

  • [MirageOS Spring 2019 hack retreat roundup]


[OCaml Planet] <http://ocaml.org/community/planet/>

[MirageOS Spring 2019 hack retreat roundup]
<https://mirage.io/blog/2019-spring-retreat-roundup>


Old CWN
═══════

  If you happen to miss a CWN, you can [send me a message] and 
  I'll mail
  it to you, or go take a look at [the archive] or the [RSS feed 
  of the
  archives].

  If you also wish to receive it every week by mail, you may 
  subscribe
  [online].

  [Alan Schmitt]


[send me a message] <mailto:alan.schmitt at polytechnique.org>

[the archive] <http://alan.petitepomme.net/cwn/>

[RSS feed of the archives] 
<http://alan.petitepomme.net/cwn/cwn.rss>

[online] <http://lists.idyll.org/listinfo/caml-news-weekly/>

[Alan Schmitt] <http://alan.petitepomme.net/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/caml-news-weekly/attachments/20190409/a0424555/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://lists.idyll.org/pipermail/caml-news-weekly/attachments/20190409/a0424555/attachment-0001.pgp>