What If Every SQL Query Could Update Incrementally? (with Lalith Suresh)

Explore how incremental SQL query updates can revolutionize database efficiency by processing only data changes, not entire datasets.

Ask about this video. Answers come from its transcript only — with the timestamp, so you can check them.

Generated from the transcript and can be wrong — check the timestamp.

Key Takeaways

Incremental query updates drastically reduce compute by focusing only on data changes.
All SQL queries, regardless of complexity, can be incrementally maintained using a small set of operators.
Feldera’s implementation demonstrates practical viability of the DBSP theoretical breakthrough.
This method bridges the gap between batch and streaming data processing paradigms.
Incremental computation improves efficiency, scalability, and latency in data systems.

What the video covers

Traditional databases rerun entire queries even for small data changes, causing inefficiency and high compute costs.
Incremental view maintenance aims to update query results proportionally to data changes rather than full recomputation.
Most current systems only support incremental updates for simple queries, leaving complex queries inefficient.
The DBSP paper proves mathematically that all SQL queries can be incrementally maintained using just four operators.
Feldera, led by CEO Lalith Suresh, implements this theory in practice, enabling efficient incremental updates for complex queries.
The approach unifies batch processing and stream processing, reducing latency and compute resources.
Incremental computation is applicable beyond databases, including network topologies and live social graphs.
The solution relies on advanced mathematical concepts like Z-sets and differential dataflow but simplifies them for practical use.
Deterministic pipelines and careful state management are key to achieving reliable incremental updates.
This breakthrough could transform how large-scale data analytics and real-time data processing are performed.

Chapters

Why do traditional databases rerun entire queries instead of updating incrementally?

Traditional databases are designed to recompute queries from scratch because changes are not treated as first-class citizens, making incremental updates complex and often unsupported for anything beyond simple queries.

What is the key breakthrough of the DBSP paper?

The DBSP paper proves that any SQL query can be transformed into an incremental version using just four fundamental operators, enabling efficient updates proportional to data changes.

How does Feldera apply the theory of incremental query updates?

Feldera implements the DBSP theory in software that allows complex SQL queries to be incrementally maintained in real-time, reducing compute costs and latency in data processing pipelines.

Full Transcript — Download SRT & Markdown

Speaker A

There's a problem with databases that has bugged me for years, and it's probably bugged the industry since the 80s, and it goes like this. Imagine you've got a database of sales figures, and you want to know sales by country. So select sum of amount, group by country, right?

Speaker A

Let's say you sell to a dozen countries, and you've made a lot of sales.

Speaker A

So that query is going to have to read 10 million sales rows in to produce 12 summary rows out. And that's probably an important query, so you end up running it a lot, but it's 10 million rows you're processing,

Speaker A

so it's an expensive query, so you end up caching it or materializing it. Here's the thing that bugs me. When you make one new sale, and it's time to update the cache or update the materialized view, how much work is that? Logically, to me, that's one operation.

Speaker A

You find which of the 12 countries the new sale was made in, you add it to the total. But plenty of systems will say, "No, no, sorry, we're not set up to do that. We have to rerun the entire query from scratch."

Speaker A

So it's 10 million and one pieces of work. They say, "Can't you just add the number in?" "No, I can't. It's against my principles. It's not how I'm designed." You have to rerun queries from scratch every time. That's horrible.

Speaker A

Now, in fairness, some systems will say, "Yes, we can add in the number. We can handle this change incrementally, but only because it's a simple query." Well, then I ask, "What counts as a simple query?" The rules to a simple query are

Speaker A

completely arbitrary, and you'll only find out that it's not a simple query when it suddenly breaks, and you're suddenly paying 10 million times the cost.

Speaker A

The incremental view maintenance problem is one we've been wrestling with for decades, and it's nasty, and it really hurts when it goes wrong.

Speaker A

And it was pretty much solved about three years ago with an award-winning paper called DBSP. And that paper proves that the answer is, all queries are simple enough. All queries can be turned into an incremental version that only has to consider the new changes.

Speaker A

I find that quite exciting, because it's like 10 million to one exciting. And the theory has been proven mathematically, and it's been proven in practice with some software called Feldera.

Speaker A

Joining me to explain how it works is Lalith Suresh, who's Feldera's CEO, and we're going to discuss why in practice it applies to database queries, but also to changing network topologies and live social graphs.

Speaker A

We're going to discuss the theory and why it works to unify batch processing and stream processing, and a lot more besides. Once I got my head around this is actually all put together. I've got to say it was the most interesting paper

Speaker A

I've read in years. So I hope we do a good job of getting these ideas into your head, and it will worm its way into your head too.

Speaker A

I'm your host, Kris Jenkins. This is Developer Voices, and today's voice is Lalith Suresh. [MUSIC] I'm joined today by Lalith Suresh, CEO of Feldera, is that correct?

Speaker A

That's correct. How's life as a CEO in 2026 for a tech company? Well, they call it a roller coaster for a good reason.

Speaker A

Yeah, I'll bet. Yeah. Although 2026 is a roller coaster, and I guess it always is when you're the CEO, you have hit upon a solution to a problem that I think I first experienced in the early 2000s, which probably dates me.

Speaker A

But it's the problem of... This is the way I think of it, right? You've got a database and you're running a view, and you want to turn it into a materialized view because that sounds like a nice, efficient thing to do.

Speaker A

But then whatever database engine you're using usually says, "Oh, I can only make it fast to update the materialized view if they're very simple queries." And that's my very vague headline, why I care, but why don't you go into the details of what you're actually solving?

Speaker A

Yeah, happy to dive into it. I think the best way to contrast this problem is to just look at how most of the industry goes about

Speaker A

large-scale data compute and analytics, right? It really, in my opinion, hasn't changed a whole lot since the 70s.

Speaker A

And in fact, the incremental compute problem, I think the first paper on the subject was in 1979 or something, so it's quite an old problem. And so things like Spark, Snowflake, Databricks, all of these are really good at what we call batch compute,

Speaker A

you throw a lot of hardware at them, you aim it at some data that you leave in your database or warehouse or lakehouse, you write SQL queries against that data. These engines will use whatever resources they have available for them to go over all the data

Speaker A

to give you back an answer. That's kind of how we understand evaluating a query or a view in the database. Now the question is what happens if you literally rerun the same query a second later?

Speaker A

They pretty much have to repeat exactly what they just did the last time, even if only five out of 50 billion rows have changed in the meantime.

Speaker A

Yeah, this is exactly the thing that was bugging me in the 2000s, right?

Speaker A

Exactly, and it's very like everyone runs into this at some point working with a database, or a warehouse, what have you.

Speaker A

And now the question is, how do you recover that inefficiency? You'll see it everywhere, and it's one of those things that once I point it out to people, it's very difficult to unsee it afterwards.

Speaker A

You look at it around you and you see this everywhere. There's a bit of a, it's quite perennial, and you leave a lot of inefficiency and latency on the table, because you need compute proportional to size of the data always. Yeah.

Speaker A

Now the incremental compute problem is, how do you turn this into a problem where the time in computers proportional to the size of the change, right?

Speaker A

By only looking at how you need to change just from one moment to the next, can you keep arbitrarily complex queries fresh at any given moment in time.

Speaker A

Yeah, if I've queried a billion rows, and got an answer, and then one new row comes in, do I have to do one computation or a billion and one?

Speaker A

Depends on the query, but ideally, you're only computing what's, like the compute is spent as proportional to size of the change, right?

Speaker A

Let's say if your query is, you know, they can be bad queries, where let's say one side of join matches a billion rows on the other side of the join, and then you are spending work proportional to a billion

Speaker A

instead of let's say one, right? But this is exactly the idea, right? Like you want to keep it as close as possible to only what change, and nothing is right. So you touched upon an interesting point when you said databases for a long, like for a while,

Speaker A

if at all, allows us only for very simple queries, and there's a good reason to it, because that generalization has been incredibly hard, right?

Speaker A

Like how do you do this for any query, right? Is it even possible to do it for an arbitrary way?

Speaker A

It's even possible, exactly, right? And so if you look at the last, again, 40, 50 years of papers on the subject, there's been some really advanced math thrown at this problem.

Speaker A

So people take a very specific type of query or an operation in SQL, and they figure out like what kind of advanced math can we do to turn this query into something that computes the derivative of the query, right? Like how does it work on the

Speaker A

changes or something like that, right? And these things have never quite hit mainstream in a way, right? So differential dataflow was a really good piece of work here, but even that's actually quite a hard nut to crack, right? Like you folks who read the paper would know.

Speaker A

And so the surprising breakthrough that my co-founders came up with was that with a very simple bit of math, just four operators, it turns out that any program written using those four operators can be automatically turned into a program that computes on the changes.

Speaker A

Now, this is a thing I always like in math and computer science when you feel like when you really hit on the nub of a problem, it bre

Speaker A

Even more powerful, and we can get to that in a moment, is that once you deconstructed it into this piece, and then you start to say, ask questions like, how do I now do joins using this formulation? You actually recover old results

Speaker A

in database literature, because the way to do joins using DBSP actually recovers the famous delta join algorithm, as an example.

Speaker A

Okay. Yeah. So this is a sign that you really found something very fundamental. Like it's actually recovering the results that others poured a lot of time into finding for like sort of specific subsets of the problem, right?

Speaker A

That's kind of, yeah, again, it appeals to researchers a bit more. Yeah, yeah. But in practice, what that means is literally you can throw 50,000 line SQL programs at us, like hundreds of views, deeply nested, thousands of joins,

Speaker A

aggregations, sliding windows, whatever. And it does not make any compromises. So it works for inserts, updates, deletes. It works for, again, any composition of these kind of queries on top of each other.

Speaker A

It even works for recursive queries. It even works for streaming problems, right? Like it just covers all of it. And at the, deep down inside, we are not special casing for any one of these cases, right?

Speaker A

Like it just works. Yeah. That's also quite uniformly. Yeah. This very much appeals that you can take an arbitrarily complex SQL statement and reduce it down to four operations. Yes.

Speaker A

And we're going to get into those operations, because I really want to break down the paper, because it's one of those papers that once the penny dropped, it starts to think, oh, actually, this is really elegant and probably more simple

Speaker A

than I expected before I started reading the symbols. Yes. But what we do, I do want to talk just a bit about how you're applying this theory, because I can see how this applies in the world of databases,

Speaker A

like make your rarely changing queries really efficient. But you were saying this applies in the world of Kubernetes and other things as well?

Speaker A

Yeah. That's actually quite fascinating, because a lot of problems start to look like incremental view maintenance or incremental computation problems, right? Like the origin of this work was Leonid, our CTO, looking into network control planes, right?

Speaker A

Okay. So there was a sort of movement called software-defined networking. These days, it's become very commoditized, I would say. But the idea is that your network in a data center, right? Like your switches, your routers, your virtual switches, the VM sitting on it, all the ports, the connections, the

Speaker A

routing tables, the rules that govern, what traffic is allowed to go where, all of that state is managed centrally using some kind of network controller.

Speaker A

And so that network controller is really what it's doing is it's like a policy, like it's evaluating policies on this sort of topology and sort of this rich graph of information. And that's really like a view. A policy is really a view over a graph, right? It says, give me the sort of,

Speaker A

give me the sets of VMs that are in violation of some kind of segmentation policy. Like that's actually the controller trying to maintain a view over a graph and it maintains a lot of lots and lots of things depending on the policies you're trying to maintain.

Speaker A

Okay. So yeah, I can see how you translate a graph into SQL and therefore watching changes on the graph becomes an incremental view problem. Exactly. Yeah, I can say that. And yeah, and so generally, sort of, if you look at these control plane orchestration or

Speaker A

cluster management platforms and things like this, typically the architecture has been, I would say, very stable for a long time. It's that you have some kind of control state database behind the scenes and then you write controllers that actuate on that state and also changes to that state. They

Speaker A

maintain views over that state. They watch the views and then they take some action against the underlying thing that they're managing. So in Kubernetes, you have controllers that watch the central database, typically etcd, but other distributions have other variations of it. These controllers then listen

Speaker A

to changes and then they take some action. They might say, you've turned on a pod or they might move it somewhere, things like that.

Speaker A

It's very similar in networking as well. They might listen to these changes. They fiddle with some routing rules, things like that. Yeah. And I bet plenty of them have lost a lot of engineering time into optimizing indexes and views and queries so that they can be rerun many,

Speaker A

many times very fast. Yes, exactly. And so the funny thing is, like, a bunch of us come from that background, including myself, right? Like my first entry point into this problem space was also cluster management. I did some work. So we were all

Speaker A

researchers at VMware before this and that's kind of where we all started working with each other. And there, there's a couple of papers I'd written on sort of using SQL-based declarative cluster management. How do you write the control plane layer by maintaining your

Speaker A

state in a SQL database? And then all the control plane logic becomes views and constraint solving problems that you build on top of that database, right? Yeah, yeah. And so when you work in that space, you very quickly run into the incremental view

Speaker A

maintenance problem as well. And so I remember, like, one of the early conversations where I had, I was talking to Leonid and Mihai while they were working on this problem, right? And I said, like, this has to be a solved problem. Like databases should have

Speaker A

some kind of IVM capability out of the box, right? And they said, surprisingly, no. No, no. And that's, yeah. I've seen some pretty large companies where they have a version of materialized views. And basically, as the query gets more complex,

Speaker A

it's more likely that they'll say, no, we can't refresh this incrementally. We're just going to re query the whole thing from scratch every time. Correct. Yeah. Or on demand, which is not actually that much better. Yeah. Which is why that you see that sort of bias towards,

Speaker A

you can run very simple queries this way. Whereas in fact, like the, the biggest value for this type of computation model is to be able to run really complex stuff, the more data you have, right?

Speaker A

Like that's when the value of the platform grows and grows and grows, right? Like, so being able to do a couple of simple queries is fine. It's cute. It's a feature. It's not really something you make an architectural decision at all. Yeah. Yeah. I can imagine

Speaker A

that generally, the more complex the query, the more it's going to cost you to rerun it every single time. And thus the more benefit you, but for that, you have to prove that your theory actually covers every kind of SQL, right? Yes. So here's the thing. So first of all,

Speaker A

the paper proves that you can cover all the SQL and extensions into this model, right? I don't think I remember the last time there was some kind of gotcha that we ran into on the SQL side, right? Like the

Speaker A

translation is the beauty of it is, completely syntactic, right? So converting into DBSP is quite straightforward. And then from DBSP into the incremental version of the DBSP program, like there is no guesswork. It's heuristic free. That's one of the keywords you

Speaker A

see in the paper early on. And that's an important detail. There is no cost-based guesswork that we're doing here. It's just complete sort of, you see, you see the program, you just mechanically turn it into a program that works on the changes. And that's it.

Speaker A

Right. So you can literally compile down from SQL, deterministically to the incremental version. Yeah. Right. So should we try and get into it?

Speaker A

Yeah, there's some creativity involved. No doubt. But where does the creativity come in? I think the SQL to DBSP. So there's a bit of, so if you read the paper, this is thing of like the actual DBSP theory says nothing about databases. All right. Right. Applying it

Speaker A

to databases is a transition that we manage by using Z sets. Right. Like applying it to the domain requires some level of creativity. And then from the SQL to DBSP layer, again, we have like a big suite of

Speaker A

handcrafted incremental operators and things like this, as opposed to writing the most general version of everything. Like there are, there is a room there, but that's again, that comes down into the level of the details, I would say. That sounds to me a bit like

Speaker A

deterministically compiling, but then there are differences between CPUs. So there's still some creative compiler works like that. Yeah. Okay. Yeah.

Speaker A

So you've mentioned Z sets or Z sets, as we say over here, I guess we'll end up using both terms. I'll call them Z sets. You call them Z sets and we'll move on. But this is, this is a fundamental

Speaker A

algorithm that no, sorry, it's fundamental data structure on which we're going to build the rest. So why don't we cut our theoretical teeth in this conversation by you explaining what they are. Sounds good. So Z sets, I mean,

Speaker A

just taking one step back from the string central to making the DBSP work sort of the map on how do you compute on streams? How do you turn a stream of changes back into the original stream? Like that's kind of the craft, that's

Speaker A

kind of the beauty of what the math allows you to do. But there is an assumption that the elements floating through the streams from what's called a group, which is a fancy way of saying there's some well-defined operations to add, subtract elements from each

Speaker A

other. And there is sort of the concept of zero value. They're associated, they're commutative. A lot of nice mathematical properties come out of having those assumptions in place. But generally databases are not groups. Right? So inserting two rows is insert one,

Speaker A

like insert a row, insert another row, right? We can do that. But generally databases cannot delete two rows of like one instance of two rows of the same kind.

Speaker A

One instance of two of the same row. Yeah, that doesn't, yeah, that doesn't make sense in database exactly. Exactly. So these are a very simple way to turn data, represent databases in a model that still fits like this group structure, right? Okay. All it

Speaker A

does is for every row, there's a weight column. And a weight column is an integer, like it's a bigger integer weight typically. So plus one means you add one instance of the row into the database. If I insert one user row, then that's

Speaker A

going to have a weight of one. When I delete it, it's going to have a weight of minus one. If the row doesn't exist, it has a weight of zero.

Speaker A

The rows doesn't exist, it has a weight of zero. But if you add a row to, so first you insert a row with plus one, then if you insert a row with minus one, it's effectively equivalent to that row only existing zero times.

Speaker A

Yeah, okay. That makes sense. Yeah. So this is what allows you to like, deal with both positive changes and negative changes.

Speaker A

Yes. Now I looked at this paper, I thought this looked to me a bit like a multi-set in which you say every row in the database, put it in a set with a value of one as its weight. Yeah.

Speaker A

And the only thing that makes it takes you to this slightly strange version of a Z set is that you're allowed negative weights to represent the row disappearing.

Speaker A

Correct. Yeah okay. Yeah, that's all there is to it. And now what this Z set allows us to do in conjunction with DBSP is that in Feldera, inputs, tables, views, outputs, changes to tables, changes to views, are all Z sets.

Speaker A

Right. Like all Z sets, even like literally the on-disk sort of journal format, everything is just Z sets throughout.

Speaker A

So in some places in the algorithm, you've got a Z set representing my user with a weight of one. Some places you've got the whole user table with a weight of one. Sometimes you've got the whole database with a weight of one, conceptually. Yeah, it's all uniform and

Speaker A

consistent and symmetric. And so this is an incredible superpower. I mean, think of your average sort of transactional database and how hard change data capture is operationally to just set up.

Speaker A

Yeah, right. It comes from the fact that changes are not first class citizens in those systems. It's not the same data structure to represent your tables versus changes to tables or changes to views, things like that, right? Like that's basically where that pain comes from.

Speaker A

In Feldera, none of that is really an issue because everything is built up in Z sets. Change is a first class citizen.

Speaker A

I'm going to hold on to that phrase. Yeah, and so some changes, and we can talk a bit more about that as well, like databases actually work on changes. We just never called it that, right? You have a database with some tables,

Speaker A

you apply a transaction to a database. That's really a bag of changes that you'd apply to the database, right? Some sort of instance, updates, and now you get a new version of the database, right? You can think of it.

Speaker A

So basically, a database problem can be represented to one where you have these streams of transactions showing up and out at the other end are coming out a stream of database snapshots, each of which represents the version with the prefix of transactions applied to it. Yeah, okay. Yeah. Applying a series

Speaker A

of transactions takes a stream of, "I want to change this into a stream of the database has changed to this." Yes, exactly.

Speaker A

Yeah okay. Yeah yeah. Like that's kind of the model. So we've always actually been operating on changes and with databases, they've just were never first class citizens.

Speaker A

And so now if you're trying to do an incremental or rather describe incremental view maintenance on top of this model, it really looks like a view is just a function of that database function.

Speaker A

Right? Okay. So when you have a stream of transactions, you have a stream of database snapshots.

Speaker A

And from that, you can have a stream of views over this evolving snapshot as well.

Speaker A

A stream of this data, the view. Yeah, exactly. And so again, so this is how DBSP ends up getting applied into this database.

Speaker A

As we said, we have this model of like databases really are streams of transactions, snapshots, streams of views. And over that, we basically find ways to do what's called differentiation and integration that is change a stream and to the, it's a stream of changes

Speaker A

in some sense. Right. So this is my impression of this whole paper is you've got a stream of transactions, you know how to apply those to make a new database, which theoretically gives you a stream of new databases. A view is just a function over it.

Speaker A

So that gives you a stream of function, a view results. Yes. That's horribly inefficient, but that's what we're doing. And if we make it maths, we can rearrange it.

Speaker A

Yes, yes, yes. So that finding that incremental query operator, the Q, turning a Q into Q delta is basically what Feldera does. You describe, you tell us if you will turn it into a Q delta. You feed us, you change, you know,

Speaker A

it's up these deletes. And what we compute internally is the changes to the views and the applied them. Yes. Okay.

Speaker A

And that's basically how all of that ties together into. For that, you need four pieces. Let's try and break this.

Speaker A

And it's my attention. I don't think we can reconstruct the entire paper in speech.

Speaker A

Probably not. But what I think we can do is explain all the pieces such that if someone were to read the paper, it wouldn't be desperately scary to look at, right?

Speaker A

It would demystify it. That's what I want to get to. Sounds good. So there's four operators in DBSP. Really, like two of them are the most important ones. Two of them are really specific to recursive queries, which we probably don't need to cover.

Speaker A

But the first two are very simple. One is the lift operator, which you can really think of as a map. Okay.

Speaker A

Over some say you map a list, you get the same list with the function applied to every list.

Speaker A

We've almost already talked about this. This is mapping the view function over a stream of database.

Speaker A

Basically. So that's what the map looks like. And then there's the Z minus one operator, which comes from signal processing. You'll see a shout out to that in the paper. But what it does is if you have a stream, it delays the stream by one time.

Speaker A

It's called the delay operator. So the element that, you know, so what this allows you to do is construct these little feedback circuits where you take a stream, let's say you add it with one time you delete.

Speaker A

And this is how you can do integration, for example. Like you can say, take the current value that just showed up and add it to the previous output we computed.

Speaker A

Right. You can, yeah, you can basically describe things like this. Basically, what happens is out of these out of just the map and the delay operator, you can now construct stream differentiation and stream integration, which is how do you turn a stream into the changes of the stream?

Speaker A

Yeah. And similarly, how do you do the inverse, which is given a stream of changes, how do you reconstruct the original stream? Right. Like that's what differentiation and integration is. So let me see if I've got this because

Speaker A

you imagine starting with an empty database. And you apply a transaction to it. Then if you route the output of that back in, from that you can pull the trick of saying, how do I have a constant input to this function, which is the current state of this database?

Speaker A

You just set up a feedback loop. You just set up a feedback loop. Right. So that's what lets us conceptually think of it this way. Like you have some delta coming in that you're trying to apply. And what you have with you now is

Speaker A

the previous state of the database. Yeah. So mathematically, you can sort of add the two together.

Speaker A

Basically. Yeah. Yeah. Okay. So let's turn this into these little math expressions where you can say like the output now is equal to the latest change that arrived plus the previous output.

Speaker A

Right. It allows you to describe these type of by induction type of formulas. Right.

Speaker A

Yeah. Yeah. Yeah. You're formulizing these intuitions we have about the way we say, and now I've got the new state of the database.

Speaker A

I want to add another transaction to that and set the differences along. Yeah. Yeah.

Speaker A

Okay. In fact, like we do this all the time in programming, right? Like you write a for loop and you say I equals I plus one, right? That's really a way of describing in some sense, take the value you have right now, add it to the previous value.

Speaker A

Yeah. Yeah. Right. Like it's really that sort of model brought into database. Right. Right. And so of course it assumes that there's some notion of what a plus means in databases.

Speaker A

And this is the small sort of technical matters of how these sets help here.

Speaker A

Right. You can add a change to a database. You can subtract a change from a database and you get the original.

Speaker A

Right. Yeah. So you can model a delete as add minus one to the state we previously computed.

Speaker A

Okay. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. So from basically stream integration and differentiation, again, constructed out of those, the map and delay operators, we can then keep going and turns out like all of SQL can be expressed as these little circuits out of just these two operators

Speaker A

pretty much. Yeah. I just want to, because there's one part of this paper where a penny dropped for me, which I thought was quite neat on differentiation.

Speaker A

It's like I can imagine having the state of my large database now and then after another transaction and after another transaction.

Speaker A

But if you gave me two database states and said what's changed between them, I think that's a really, really hard thing to compute.

Speaker A

Right. And yet with Zed sets, it seems to be suddenly very easy. Yes. So part of it is you have to look at it as you're getting to this, you're always spending a little amount of work to get to the final answer.

Speaker A

Right. So if the two big databases showed up at once and you asked to compute the diff, you really do have to spend the time to actually at least scan both of them. Yes.

Speaker A

If you gave me like two Postgres databases and said spot the difference, that would be a really hard algorithm as well. Yeah, I think so.

Speaker A

You kind of have to, you basically have to index both sides in order to compute that.

Speaker A

But you've pulled a very neat trick in DBSP to solve that, right? You just say, since both databases have overlapping data, but different weights, you can just sort of take the difference of the weights.

Speaker A

You basically can add them up actually. Yeah, you can add them up and you get the reconstituted. Because they're weighted sets, you just do one set minus the other or plus the inversion of the other.

Speaker A

Yeah, there is something like, again, depending on exactly the SQL semantics, except versus intersect have slightly different meanings, right?

Speaker A

But yeah, you can, you basically can just add up these sets in order to do some of this.

Speaker A

Okay, so you have the ability to roll changes into a state and figure out the difference between two states and thus get a sequence of the changes back out. Mm-hmm, yeah.

Speaker A

Delay a thing and lift a function into working on changes instead of individual states. So those four?

Speaker A

Not quite, right? So one thing is a lot of computations, when you turn them into the sort of model where you work on changes have to be stateful.

Speaker A

And all state in DBSP or Feldera is packed into those z minus one operators.

Speaker A

Got it? Right. And sometimes you can think of those, the stuff that we keep behind there, that's the only state that we need to maintain on storage or whatever, the only thing we need to check point. It's also what you should think of as

Speaker A

indexes that we keep for the computation that you're running, right? Okay. Yeah, so anytime there are joins, distincts or aggregations, or certain kinds of aggregations, actually, these become cases where they have to be stateful, which means we have to keep indexes

Speaker A

on all these things, right? And really fundamentally what it means is there's a z minus one operator in there. Right.

Speaker A

Right, yeah. So this is where that's coming from. Yeah, because that's what you're doing, right? When you do a delay on a thing, you're saying keep hold of the previous version. Correct, correct.

Speaker A

Yeah. Right. And there are operations which are linear operations, like filters or projections, these don't need any state. Yeah.

Speaker A

They're also linear aggregations, like sum, for example, right? Sum and count, like they also don't need any state.

Speaker A

You can, what this means is by only looking at the delta, you can compute the delta to be just one number to the old number, but you've got to recalculate the whole table. So you're saying certain classes of changes, we know how to do those incrementally.

Speaker A

And when one change comes in, we know how to deal with that change on the old state.

Speaker A

Yeah, exactly. Without keeping any sort of complex memory or state of the past version of the database or anything.

Speaker A

Okay. But there are computations where you just can't escape it, like joins the things. Let's just take off the easy ones.

Speaker A

So that's got to be sum and count. Average is just sum and count. Min max. All of that, yeah.

Speaker A

Selecting. Min max or not? Min max or not? Oh, really? Yeah. So in the general case, when there are updates and deletes, they're not. Oh, deletes.

Speaker A

Right? Like yeah. Yeah. Do you think have to hold across again, like we're not playing terms and conditions games.

Speaker A

Right? Like, so the general version of it for min and max, you do need state. They're not linear aggregations.

Speaker A

That said, if you have append only streams, that is, there's no deletes or updates, they do become linear. That makes sense.

Speaker A

They only need the past version. Because if I insert one, two, three, four, five, then delete five, four, and three, you've got to have had the information available, right?

Speaker A

Correct. Okay. Okay. So I don't get those for free. I get selecting columns out of a table because I can just look at the new columns for the new row, right?

Speaker A

Correct. Yeah. Filters. Filters. Yeah. Yeah. That's also. Yeah. Those also don't need any state.

Speaker A

They're sort of naturally incrementable. Incrementalizable. Yes. Okay. Group by join distinct. Yeah, group by, it's also stateful, right?

Speaker A

But yeah, joins distinct aggregations. Certain kinds of aggregations. Group by things like that actually require the delay operator with the state, and therefore, we will keep state on it.

Speaker A

Okay. Take me through one of the harder ones then. So if we can tackle that.

Speaker A

I would say the join is a good example for this, right? Since I said we do recover the delta join algorithm.

Speaker A

So what is a, again, it's hard to draw circuits here, right? Take a look at that particular figure in the paper where it describes the circuit for the join. You should view it in this lens.

Speaker A

It's basically like what is the change to a join that you're trying to compute, right? Like say there's two sides to the join, the left and the right. You need to find all changes to the left relation.

Speaker A

Find out like, so let's say there's a new change you're computing. And so you want to compute the changes from the, you take the changes on the left relation. Join it against the right side.

Speaker A

So it's delta A to join B and then the opposite of it, delta B join A.

Speaker A

And then add it to the old value of the join, which is A join B.

Speaker A

So you're really just computing for both sides what changed that would now affect the output of the join. But then you have to add those z sets to the old value of the join. That's it.

Speaker A

So when something new comes in, I'm saying, does this new thing suddenly now joined something on the other side? Is there something on the other side now?

Speaker A

Yeah. I'm trying to break that down to words. I join user to account. I get a new user in, I've got to scan to see if there are any existing accounts that would affect.

Speaker A

Yes. If I get a new accountant, I've got to scan for any existing users. Yes.

Speaker A

And if I get both in, I've got to see if they interact as well.

Speaker A

Yeah. Right. I think I get that. Yeah, that's it. That's okay. And that in a nutshell is when delta join.

Speaker A

All right. But the way you construct joins out of DBSP will basically recover that algorithm out of the box.

Speaker A

Quite naturally. So with those operators in hand, let's go back to the original one. I've got a series of transactions coming in.

Speaker A

They go to a database. Yes. I end up conceptually with a stream of what's changed in the database.

Speaker A

And I run my function, view function over that. I get conceptually a stream of what's changed in the view. Am I eventually then going to have to roll all those view changes up to get the materialized view that people actually want to query? It depends.

Speaker A

You don't have to. But so in fact, let us, for example, we distinguish between like regular views, which are just computing and pushing deltas through materialized views where we do reconstitute the entire tables. And then we have this special thing called

Speaker A

the local view, which is views that are not externally observable, which we can do some really aggressive global optimizations over. But that's kind of in detail.

Speaker A

But like there is a, we do distinguish between just propagate delta. For example, in a create table statement in Feldera, where you hook it up to, let's say, Kafka or whatever, some kind of input connector.

Speaker A

If you don't have primary keys, or if you don't declare it as materialized, it will not use any state. It's just shoving deltas through it.

Speaker A

Good. Yeah. Right. Do you, are we just talking about data definition? Correct. Sorry. Are we talking about DDL or DML? Are we just talking about inserts, updates, deletes, or are you also talking about alter table, create table?

Speaker A

That's a good question. No. We like, so generally we assume the schema is fixed.

Speaker A

That is the create table, create views, and then we have to do that. Create table, create views, statements are fixed. These are standing queries.

Speaker A

You have inserts, updates, deletes coming in. We also do allow you to sort of change the views, right? Like you can change your pipeline.

Speaker A

You can sub out some of the views for other views. You can add new views.

Speaker A

You can move views. And we have mechanisms to actually compute the new views from the state you already have as well.

Speaker A

So the, one customer called it the incremental, incremental computation or something. Like yeah. Yeah.

Speaker A

Yeah. Yeah. Yeah. I wonder if any of this theory has been applied to the problem of schema migrations.

Speaker A

Schema migrations would, yeah. We don't handle it out of the box, but it naturally does fit this model. Oh, yeah.

Speaker A

Okay. So typically like there are cases where you can't avoid a backfill, right? So if you had a hundred tables and you remove one of the tables, for example, right? Or you add a bunch of columns that you

Speaker A

didn't have before, things like that. There's a bit of a policy decision or whether you want to backfill completely or you want to, you know, just assume nulls, recompute what you had before, but we for today do require you to backfill just that table.

Speaker A

Okay. So how does this end up getting used? Is it like, is it backed by some regular database or is it integrated into an existing database? Or is it entirely standalone new kind of database?

Speaker A

Or is it attached to Kafka to get like a historical record of stuff? Standalone. So we've written this query engine from scratch, like I said, because changes are first-class citizens. So it's very difficult to, it's very difficult to bolt this on an existing platform that doesn't

Speaker A

have a native understanding of what changes are, right? So we wrote the entire thing from scratch. We have our own storage layer optimized for dealing with things like Z-sets, all kinds of operators to work on different kinds of SQL operations. We have a SQL compiler we wrote and

Speaker A

an entire control plane around it. Right, yeah. Yeah. And so the model is users describe SQL pipelines.

Speaker A

A pipeline is just a bunch of create table and create view statements. Tables can be hooked up to input connectors, like zero or more input connectors.

Speaker A

Views can be, you can have output connectors attached to views. You can have zero or more output connectors per view.

Speaker A

These connectors can be any combination of live or historical data, right? Like it can be any combination batch of streaming data source or destination, right? So you can have, let's say, get your historical backfill from some kind of Delta Lake snapshot

Speaker A

and switch to Kafka for live traffic for this table. Okay. And you can have a very heterogeneous mix of traffic going in and out. Because you've got a backing theory that says, if I've got a stream of changes coming in from Kafka,

Speaker A

I can reconstruct that into a database-like thing. And if I've got a constantly changing database, I can reconstruct that into a stream-like thing. Yeah, yeah.

Speaker A

They're all changes to us at some level. Like a batch of, let's say if you do a backfill from S3, that really just looks like one big batch of changes showing up at once. Right, yes, because there's no

Speaker A

difference in your theory between one change of one row and one change of five rows or a million rows. Yes, exactly.

Speaker A

It makes absolutely no difference for us. So the computational model is, we call it the synchronous computing model, in that you take a batch of changes at the input, you evaluate all of it, and you compute the set of outputs.

Speaker A

That changed, and this all happens atomically. Right. Right? Yeah. So connectors streaming a batch or what have you, or whatever combination of, their only job is to interface with the outside world and supply some set of changes to

Speaker A

apply in the next round that you run. That's all they do. So it's fairly cleanly decoupled. In fact, it provides a nice boundary between the messy outside world and this nice clean internal set we have.

Speaker A

And then similarly on the output side, it's very similar. The engine provides a set of changes that they connect to ship to their destinations, and that's it. It's not fun.

Speaker A

Fundamentally, if the key to this is getting a concept of changes to be first class, and then it seems like a lot of nice things fall out of it, why do you think it took this many decades to solve the incremental view maintenance problem?

Speaker A

It's hard to sometimes answer these questions because it's one of those things that really good breakthroughs look obvious in hindsight, but getting there is actually quite hard. This was one of those things where really, really, really, really smart people worked on it for

Speaker A

very long time. If my co-founder said the math that came before this was too hard for them to understand, I believe them. So it just took, I would think, an outsider lens of the problem to basically crack that code.

Speaker A

And so bringing digital signal processing into this mix was, I would say, a leap of creativity, I think, from Mihai to pull this off.

Speaker A

Because a lot of this theory comes from how do you compute functions on audio streams.

Speaker A

Yes. Yes, I think I did. The operator actually comes from digital signal processing, the Z minus one operator.

Speaker A

Right. So the same math that's trying to do chorus pedals and echo effects. I'm not a signal processing guy.

Speaker A

I barely scraped through that class in undergrad. But Mihai does have a background in it.

Speaker A

I think he jokes that he finally got to apply his degree in DSP into the real world or something like that. That's cool.

Speaker A

That's cool. Not the first guess or connection between programming and audio that we've had in this world.

Speaker A

I guess the maths for audio processing gets really hard too, right? Pretty sure. Again, far beyond my ability to process for sure.

Speaker A

Okay. Yeah. So what makes this hard in practice? Because I can just, like, I can see the four fundamental operators are simple. I can start to see how the paper's built up.

Speaker A

I've been chewing away at this paper in the background too. Okay. I can see that writing an SQL parser and compiling it down to these maths operators is a lot of work.

Speaker A

Having an API that slurps changes from S3 and Kafka and Postgres and, and, and, and, and, that's a lot of leg work. Is it just a tremendous amount of leg work to get this into production or is there other stuff that makes it hard?

Speaker A

I think yes and no. It depends on kind of the use cases and scale you're going to. Like for us, our customers typically show up and replace these gigantic workloads running on Spark, Snowflake, Databricks with us.

Speaker A

So the bar is very high, right? Like anyone can do this at very smallish scales in memory and whatnot. Like that's easy.

Speaker A

We're not even, I typically am not quite interested in that space, right? Right. Doing this at scale, like, you know, try evaluating, you know, 3000 joins, 100,000 lines of SQL, you know, like that's where it gets, that's where this gets really interesting, right?

Speaker A

Because if you think about the value proposition here, it's really the speed up and compute savings is unbounded, right?

Speaker A

Like so the, if you think of like a little graph where, you know, you have data volumes on the x-axis and you have the time it takes to run a query the old fashioned way or compute it takes to run a query the old fashioned way.

Speaker A

Like this number just grows up. The more data you have, the more expensive batch compute gets. Whereas typically the size of the changes are minuscule relative to the overall dataset size, right?

Speaker A

Yeah, most batch jobs are just re-computing last night's changes to three years worth of data. Exactly.

Speaker A

Like, and so if you draw a flat line, that's what actually incremental compute can be for you because the changes are so small.

Speaker A

It really starts to be independent of how big your overall dataset size is as you group, right? And so the further along your, the more bigger datasets you can take on this way, the more dramatic the speed up keep

Speaker A

getting, right? Like we've seen folks with like, I think our record so far is 48 hours to two seconds at 50 x scale.

Speaker A

Like that is kind of our record. Hitting. No, that's actually the sort of team record so far.

Speaker A

But in general, like most customers, I would say two to four hours, if not nightly jobs, things like that become milliseconds, seconds, that's kind of the transition you do.

Speaker A

But with the caveat that you don't have to spend a fortune to get there, right?

Speaker A

You only need like a fraction of the compute to keep something fresh all the time. Yeah, I suppose if the query will allow it, you're not actually maintaining a lot of state to process the changes, right?

Speaker A

We do have to maintain mostly than usual because at the end of the day, what we are doing is trading our time for space, right?

Speaker A

We are keeping these indexes so that you can compute efficiently. But in general, like the compute savings and the freshness matters enough that like, it's worth it. We save so much on the compute side, like storage needs to be cheaper, so it's okay. That's at least how we look at it.

Speaker A

You've just made me think of another part of implementing this in production that must be hard, which is you've got to worry about checkpointing and state management. Checkpointing, state management, fault tolerance.

Speaker A

And of course, like the thing I think we spend a lot of time on is the backfills.

Speaker A

That is the Achilles heel here, right? Which is how do you run that first query?

Speaker A

Yeah. It's like the equivalent of the first query, how do you run that? Because there's no shortcut there, you just have to process the entire input at some point.

Speaker A

And here we are pretty differentiated in that we. Like I said, there's no distinction in Feldera between batch and streaming, right? So the backfill is really like we can actually run it as this one gigantic transaction up front.

Speaker A

So we don't end up doing a lot of intermediate work computing many small changes. Think of an aggregation, for example, right?

Speaker A

Like if you process the entire input, like let's say 50 billion rows, one at a time, you might have an aggregation, let's say the total number of users.

Speaker A

It just keeps going from one to two to three to four to five or whatever, each of which is a delta for whatever is sitting downstream, which also needs to get processed. Doing that, what you can do is we call

Speaker A

it huge steps for efficient backfills. You basically run the entire input backfill as one gigantic transaction in Feldera, which means that each operator pretty much computes on this gigantic batch of changes.

Speaker A

And they produce a gigantic delta for whatever is downstream, but you only do that say once.

Speaker A

Yeah. Yeah. Okay. Everything's changed, but some changes are larger than others. Correct. Exactly. Right.

Speaker A

Every, yeah, because of that, there's no, in the engine, there's no, it's not like a separate mode or anything like that, right?

Speaker A

Like I think conventional wisdom has been then batching streaming a completely separate lanes. We don't make that trade off here.

Speaker A

Like it's just one engine that flexes between. So compare this for me to something like Kafka Streams, which is kind of takes them a different approach, which is you've got a long stream of changes, just roll this state machine over it.

Speaker A

I would say the distinction is the complexity of workloads. You can take over it and what the semantics are. Okay.

Speaker A

So Feldera does not change SQL semantics in any way. In fact, like you can run those crazy workloads I mentioned, right?

Speaker A

A hundreds of thousands of lines is equal and row by row, cell by cell, we guarantee you get exactly what you would have gotten in a batch engine.

Speaker A

Okay. You will never see a state that is inconsistent in Feldera in that sense. Whereas my understanding of a lot of the sort of Kafka Streams, ksqlDB, Flink, like that universe of stream processors has been, they've mostly operated on this

Speaker A

eventual consistency model, right? They have very low level API, so you can think of them a little bit as assembly, but how you construct more complex operations out of it has usually been like a hard ceiling that is not at all trivial. And in fact, like performance, once

Speaker A

you get out of this interest of settings, starts to fall off very quickly. So that thing of like, can you take on the same complexity as we would with a Spark?

Speaker A

While you still operate on Deltas has been a bit of a holy grail. No one has accomplished them. We think we're pretty unmatched there.

Speaker A

And I think DBSP is a superpower that allows us to get there. Okay. Okay.

Speaker A

Okay. So what about things like user defined functions? To what degree can you support arbitrary code coming into your SQL queries?

Speaker A

We can't in the sense of like one requirement for anything you run inside a pipeline is that it has to be deterministic.

Speaker A

Okay. So only pure functions. Yeah. So UDFs, we have UDFs and UDAF. What we do is we take the SQL program, we compile it into a Rust binary that uses the DBSP crate underneath.

Speaker A

So UDFs that use UDFs or UDAF that you write in Rust are also compiled into the same binary. But if these things have a random number generator or they go, they show an external API call, which is not deterministic, things like that, that's all.

Speaker A

Okay. Yeah. Does that-- We do require you to keep them deterministic. Reminds me, I saw a company recently doing Rust where they had a similar restriction.

Speaker A

And what they did was compile it down to Wasm because they could ensure that there was no-- like the execution rules for that Wasm sandbox.

Speaker A

Like it couldn't execute impure functions because they didn't supply the means to. I can see possibly why, right? Compiling to Wasm seems like a pretty big lift, I would say, right? Like we just put the big sticker out

Speaker A

there and we use it at your own risk. It's usually for very advanced users anyway.

Speaker A

Most people don't need it. We also-- the other thing is you can also-- what you can do in Feldera is you can turn some of these impure things into inputs to the pipeline, which as it should.

Speaker A

In fact, like now, the now function that gives you the current clock value is really just another input connector. Take me through how that works.

Speaker A

So a clock is really just a stream of time stamps that comes in from the outside, right? So anytime you're basically writing a pipeline with a now function in it, again, it depends on where you apply the now, right? But for example,

Speaker A

if you say select star from some table and you say select star, comma, now, what you're really saying is at every time step, this view changes.

Speaker A

Like every record that you compute is now going to change. So that's a bit of a footgun, whereas there are cases where you can use it like as sort of as part of a filter expression, where it becomes what's called a temporal filter. And we have very optimized data

Speaker A

structures handling that efficiently. So we do warn you to use now inside a projection, which is kind of an anti-pattern. But a now is basically another, you can think of it as the now tables, which has one value,

Speaker A

which is the current value of the clock that you're really joining against. So that's how you should want it. But how would you make that work when you start backfilling? Because the now table is going to be completely different. That's correct. Yeah.

Speaker A

So you're just... When you backfill, you're really starting from scratch anyway, right? Okay. But if you checkpoint and then you recover, what happens is up to the checkpoint, everything is correct. But then when you resume from the checkpoint,

Speaker A

the now value will look different. That's all. So there's sort of a time since you recovered.

Speaker A

Correct. Exactly. If you rerun a pipeline, you're kind of implicitly saying with now that you're okay with the value changing, right?

Speaker A

Yeah. It's not meant to be deterministic at that point. But it's given the same inputs, it will always compute the same outputs.

Speaker A

We never relax that, I would say. So for testing and things like this, there are ways to actually feed the now stream.

Speaker A

Right. Yeah. Yeah. I suppose it would be semantically weird to say now means now except when we've decided that actually means two hours ago because the system crashed for a bit and we're trying to catch up. No, no, no. Yeah. We don't get into that

Speaker A

mess. It's just a stream of values for us. Yeah. Right. In fact, a bunch of users also do this thing where you can...

Speaker A

Yeah. Like the time stamps come from the data itself, right? So they might have a table, they might have time stamps that they planted on those rows ahead of time. Let's say it's already in their Delta tables or something like that.

Speaker A

And then you can just operate on those time stamps inside the pipeline and again it stays deterministic.

Speaker A

Sure. That makes sense. That makes sense. Yeah. Yeah. Yeah. So how long does it take to go from the paper working out the theory?

Speaker A

To your first genuine against production SQL queries working problems, right? What's that path like? I'd say. So, I mean, we were founded summer 2023. I would say we built it for a year and a half before we felt this was ready to actually take to market,

Speaker A

right? Like it hit certain milestones of like, do we have our own storage engine? There's a bunch of things we were waiting to fit.

Speaker A

You could already run a lot before that, but then last year is when we really took this to market, right? So that's about the timeline it took us.

Speaker A

But again, it's never done, right? You're constantly. You're constantly adding capabilities. You're constantly improving your coverage of SQL constructs. And it's usually not even things specific to DBSP.

Speaker A

It's just making sure the front end supports things users want to do, right? Like we have the single largest contributor to Apache Calcite, for example, which is what we use as our front end.

Speaker A

We've added things like lateral column aliasing, which is a construct people really like from Spark SQL. But I think it's Databricks SQL. We added that, but that has nothing to do with DBSP, right? That's just a purely front end capability. But

Speaker A

there's a lot of work in this shape that keeps showing up. And then optimization is a constant for like query optimization, right?

Speaker A

There are better ways to run some queries than others. And again, that's also very front and heavy work that we do. So this is optimization. Give me an example of when you need to optimize that's not taken care of by

Speaker A

the theory underlying it. Well, that's just a standard query optimization, right? Like things like predicates get pushed down. Oh, okay.

Speaker A

Yeah. Yeah. Like, yeah. All of those things. And there's always going to be some case that shows up where you're like, this could benefit if we applied this kind of query optimization.

Speaker A

Yeah. Yeah. All the standard SQL things apply. Like we also do this thing for like sharing indexes, right? So if you have a pipeline and you have like a lot of use, they shouldn't all be constructing their own index for similar

Speaker A

joins, for example. Right. Yeah. Yeah. They could all be sharing the exact like one index. And we've seen customer workloads where like, there'll be like 30, 40 different views that without that optimization would have each kept their own index for some part of the join.

Speaker A

Whereas all end up sharing like one index over the input tables or something like that. And these are the kinds of things that we constantly improve. Yeah. Yeah. It's not hard to imagine a company has 40 riffs on the same basic view. Yep. Yeah. Okay. Okay.

Speaker A

So if I, is I've been in the background, I've been trying to wrap my head around this paper and I've been implementing in Haskell, and I've probably been doing it the hard way to be honest.

Speaker A

But the intellectual curiosity got the better of me. If someone wants to like experiment with this, there's a paper, there's a Rust library, where would you recommend getting started? So try.feldera.com is free for everyone to use. You should start there. That's a sandbox. There's also the open

Speaker A

source version that you can run on your laptops, wherever you want, right? Like it's MIT licensed, go wild, right? Is that the Rust library? Or is that something else? It's the open source version of Feldera.

Speaker A

So it's not just the Rust library. So you can actually write the SQL pipelines. You can define connectors, all of that. So typically users start there before they want to go more serious and use the enterprise machine. Take it more

Speaker A

serious down into the lower level. It's when you really want to run this infrastructure in a scale, right? Like that's usually like, and you kind of want someone to answer the phone when things go wrong.

Speaker A

Right? Like, yeah, that's basically the cutoff to the enterprise edition. Typically things like fault tolerance, things like that are packaged into the, it runs in Kubernetes. There's a bit of, right? Yeah. Yeah. Yeah. All those, we've got the thing

Speaker A

working, but now we have to live with it. Thing is a good place to charge money, right? Yeah. Going from individual to the company needs kind of like the things that separate the two is where the open source versus enterprise edition. But there is also the Rust

Speaker A

library. If I wanted to just connect random things to random things incrementally, right? It's very low level. I think the, right? Like, so the DBSP crate, like you're not defining SQL or anything anymore, right? Like it's very low level. So we compile

Speaker A

into a program that uses the DBSP crate, but also the other things around it. So it's not just one crate even there's a lot of things that get assembled into the pipeline. That's basically where it comes in. And then there's the whole control plane thing of like orchestrating,

Speaker A

provisioning pipelines, resource management, all of that, right? Like that's the other parts of the platform. Right. Okay. Yeah.

Speaker A

And if I want to wrap my head around the paper, how should I have started?

Speaker A

I'd say his talks on the subject are amazing. That's where I would start. So, so there's a talk he's given at Confluent's conference, Current, in 2024. That's a pretty good, I think it's called streaming queries without compromise. That's a very good presentation by him. I would

Speaker A

start there. There's a talk that he's given at the CMU database group, which is like really a deep dive on the paper itself. It's about an hour long. That's another thing I would search for. Okay.

Speaker A

Yeah. Okay. And he's a fantastic speaker. If you haven't listened to him already. I haven't. I will do. And I'll link to those in the show notes. And maybe I should have done that before I started asking Claude

Speaker A

to explain mathematical symbols to me. Well, that's one way to go about it. Well, you can ask Claude again, very 2026.

Speaker A

Yes. On that, I think we've got plenty of brain food to be carrying on with. So Lalith, thank you very much for taking me through that.

Speaker A

And yeah, I feel like I've got a stream of changes to go into my brain.

Speaker A

In fact, a slide in his presentation on this is exactly how we as people process the world around us, right? Like we basically, you know, integral of the stream of our experiences day to day, right? That's yeah. Yeah. Yeah. Database is a

Speaker A

very similar. That's, that's how they work. Yeah. Thank you for having me. Pleasure. Awesome to be here. Thanks very much. Thank you, Lalith. So lots of links in the show notes for this one.

Speaker A

You'll find Feldera, the company and the open source edition. If you want to go and use it, there's DBSP, the lower level Rust library, if you're interested in diving into the code. And of course, there's the original award winning paper. Before I researched this,

Speaker A

I didn't know that papers won awards, but apparently they do. And it won one and I'd say it's well deserved. So congratulations to the team that wrote it.

Speaker A

We'll be back as soon as we can with another interesting voice from the software world. So please do take a moment to like this episode, maybe share it around and make sure you're subscribed to the channel. But until next time, I've been your host, Kris Jenkins.

Speaker A

This has been developer voices with Lalith Suresh. Thanks for listening.

Topics:incremental view maintenanceSQL queriesdatabase optimizationDBSPFelderabatch processingstream processingdifferential dataflowreal-time analyticsdata pipelines

Get More with the SozAI App

Transcribe recordings, audio files, and YouTube videos — with AI summaries, speaker detection, and unlimited transcriptions.

App Store Google Play

Or transcribe another YouTube video here →

Free tools: TXT to SRT · SRT Validator · Merge SRT · Subtitle to Text · All tools