CS Talk
Joseph W. Cutler
https://www.cis.upenn.edu/~jwc/
Host: Ruzica Piskac
Title: The Essence of Structured Streaming Computation
Abstract:
Many important computational tasks are just stateful stream processing, if you squint at them the right way: network hardware programming, ML model training, and distributed data analytics can all be framed as computations which accept input events, produce output events, and evolve some internal state. The most straightforward way to program all of these tasks is on an event-by-event basis, with the program directly describing what to do with every event that arrives — this is of course cumbersome, challenging, and error-prone. Because of this, researchers working in all of these domains have built domain-specific languages — P4 for software defined networking, and Spark/Beam/Flink/etc for ML and data analytics — which allow for programming with streams as structured objects, rather than as handles to some incoming next event. In this talk, I’ll present a new unifying foundation for high-level, structured streaming computation. Drawing on the theory of ordered substructural type systems, I’ll describe a type system and semantics for a core calculus where terms are stateful stream processors which run incrementally event-by-event, but are written as high-level functional programs that transform variable-size extents of streams rather than individual events. Lastly, I’ll touch on an extension of this type system to one which allows for programming over streams with nested parallel-and-sequential structure, instead of just sequential structure.