00:00Hi, I'm Tim Berglin with Confluent

00:02Today I'd like to talk to you about Apache Iceberg

00:04Iceberg is an open table format

00:09What is an open table format? Why do they exist? How does Iceberg work? I'd like to start, as always, with a little bit of history

00:16Let's go back about 35, maybe 40 years to the data warehouse

00:23Data warehouse was a big database that contained the data from a bunch of, well, smaller databases, right? These were the operational, usually relational databases out in the enterprise, and we'd collect their data overnight through a process called ETL: extract, transform, and load, and load it into the Data warehouse, which was then available for reports, analysis, querying the next day

00:50Uh, this was a batch process

00:54Uh, there's, you know, a whole overnight kind of thing, and it might seem old-fashioned these days, but this was great technology in its day

00:58Uh, it lasted until it didn't, eventually started to give way for reasons of scale and other things to, uh, another thing maybe 15 or so years ago called the data lake

01:11Now, it wasn't called the data lake right away

01:14It was called Hadoop at first, really, uh, but this was, uh, we'll say a big distributed file system where we would still have these

01:25Operational databases out in the world, those have never gone anywhere, right? You still have all these data sources, and maybe now mobile devices and sort of early streaming data sources and all these things, uh, coming into this data lake

01:37And instead of all of this attention to the schema of the inputs and the specialized, specially designed schema in the data warehouse, uh, and this extract, transform, load, we just kind of extract the data, load it into the data lake, and transform it later

01:51So there was this extract, load, transform, and this was supposed to be a revolutionary change, uh, to get all the data

01:58Into this, this data lake, as again, initially this took the form of Hadoop

02:03These days, the data lake, as a going concern now, um, is often, uh, cloud blob stores, say S3

02:08So, uh, this is a little bit anachronistic for the beginning, but these days these are just S3 buckets, uh, all the data from out there, bringing it into here

02:20And one of the main transitions, apart from scale, trying to make a system that can scale to larger volumes of data, is less attention to schema

02:29So there was this, this idea that maybe somehow schema was a bad idea

02:36We didn't, we didn't want to spend so much effort on that, which is completely

02:41Understandable for the thinkers and builders of the time, because schema is a tremendous pain, right? So we can kind of understand how they got there

02:47However, over time, schema does kind of matter

02:51We still do want to know the format of the data in here

02:53We still want to think of data as tabular

02:59Uh, we want to run SQL queries over data

03:01So, you know, we kind of do need some kind of system that gives us access to schema

03:07Iceberg, in particular, an open source project that emerged from Netflix, uh, they also had a concern with consistency

03:17So this being essentially a large distributed file system, I could have Little bits of tables and different files all over

03:24If I'm going to modify something, it's difficult without some kind of layer to wrap it to have a consistent view of the data in there

03:31And we would also love, uh, transactionality, that would be nice too

03:36Uh, there are other benefits, but kind of broadly speaking, in this data warehouse to data lake transition, some of these things got lost, and, uh, that was difficult, and so Iceberg and others like it emerged

03:43That's the history

03:50I want to give you an idea how it works, and I'm going to kind of start bottom up and build things a piece at a time into, uh, sort of a logical

03:57Architecture of Iceberg and give you an example of how this gets applied in the modern streaming world

04:03So, let's just say we're going to start with some data files, and these data files here, I'll just go down here, make this little document here, right? There's a file, uh here's another file, got some stuff on it

04:20Uh, we'll say these are parquet files

04:22That is certainly standard, that's a non-controversial choice, typically, the file format you're going to find in any kind of data lake

04:28So, let's just start with that

04:29We've got parquet files

04:31Let's just say I've got some, uh, we'll make it easy batch

04:33Ingest process, I got a bunch of stuff

04:35I need to get it into my data lake

04:37I could just split that up into parquet files, put them in some directory and say, "Okay, there's my table." And that is kind of like the early version of of what a data lake was

04:47I'm not going to have any schema management, I'm not going to be able to do consistent updates

04:52Certainly, I don't have transactionality, so that's not quite enough

04:56Let's build some layers on top of this

04:58All right, and I'm, I'm even, I'm going to draw a dividing line here

05:01So this is my data layer and up here this is going to be my, my metadata layer, to begin

05:11With, I'd just like to have some record of which files I've got here

05:14All right

05:17So I'm going to make a manifest file, and that manifest file is just going to say, hey, look, these two parquet files, that's your table

05:24If I'm smart and I want to be able to optimize queries, there's other things I might be able to add into that manifest file, not just a list of paths of parquet files, but these parquet files, um, you know, that's a, that's a column or data file

05:39There is sort of typing and metadata built in there

05:40I've got columns

05:41Those columns have data types

05:43If they're numbers, I might be able to track things like

05:46The max or the min value and other interesting things about this data that might help me process queries later on

05:54So, this isn't just a list of files

05:56There's a little bit more smarts to it, uh, than that here

05:58But now that I've got some collection of files, I say, well, I would like, I would like a manifest file to to to describe this ingest that I've done in in this current version of this table

06:09Life goes on though, and I do some more ingesting, right? So let's say, uh, you know, I, I write a few more parquet files down here and that's, uh, also, you know, I really would like them to be considered to be a part of the same

06:17Table

06:24So I'm going to have to make, you know, another manifest to indicate that those are there

06:31But now that I've added the ability to ingest more than once, which is pretty important, particularly in case of streaming systems that involve Kafka, I might be doing ingest kind of constantly, right? So I need to be able to ingest often

06:45Um, I've got multiple manifest files, so I'm going to need another layer on top of this

06:53We'll call this the manifest list

06:57So now I've got a way to collect multiple ingest events, or multiple manifest files, into one collection

07:07And now I can say, all right, I got this

07:08Manifest list

07:10Now, this represents the table

07:11If I want a description of what the table is, uh, go to this thing and that tells me where all my data is, tells me a little bit about it, maybe something about the schema, I can kind of pack all that stuff in here

07:22And this is pretty cool so far

07:24This is like a sane way to manage tables

07:26But what about, uh, what about consistency? What about transactionality? What about when I say I want to change the schema? Now, as I like to say, there's, there's nothing that makes schema migration really pleasant, but we would like tooling that makes it at least possible

07:40And so far Here, if I was going to change schema, I would be modifying manifest files, maybe building out new parade, new data files, something

07:50There'd be a lot of things going on, and this table would be in an inconsistent state for potentially a long time before that change were complete

07:58I wouldn't like that

08:00So maybe let's come up with a way of snapshotting, and this will be our actual metadata file

08:07This will eventually point to the manifest list, but I'm going to skip a little bit

08:11I'm going to say this is going to have a notion of a snapshot

08:15So this is snap zero right here, and that snapshot points to An actual manifest list, which points to actual manifest files, which points to actual parquet files

08:28And now, by the way, as I've built pieces here, I said before this was what our table is

08:33Now I'm saying, okay, the metadata file, this is what really describes a table

08:37It points to a manifest list, which points to manifest files, which point to data files, and it contains these snapshots

08:43So I can make snapshot one

08:46I can make snapshot two

08:49It can contain multiple snapshots, which, let's see, this might point to another manifest list, which might have another manifest file, which might point to this parquet file, and Potentially even a new one down here, and maybe even parts of this new snapshot are old data that was already in that table, since we're adding on to, potentially adding on to this table, or keeping old parts of the table as we move forward in snapshots

09:28So with this metadata file component now, finally, I've got some way of seeing consistent views of the data, even as the data is changing and as the schema is changing

09:35And that's pretty cool

09:42And that's all built essentially with parquet files and little pieces of JSON, uh, scattered in a bucket in S3

09:50Just this structure, uh, you've got a lot of this

09:54Functionality

09:56One more component we'll put on top of that

09:58We'll just call this the catalog, which, in summary, is going to tell me, you know, my table

10:05Say the table is named thermostats or something like that

10:09Well, okay, that thermostat table name, that's this metadata file, and that tells us where to go from there

10:16So, it's my way of looking things up in my Iceberg data

10:22So, that's the basics in terms of logical architecture of Apache Iceberg, and in terms of physical infrastructure, when I'm trying to learn a new system, I always like to know, like, okay, what is it? You know, what are the pieces? It's made out of? The logical structure matters, but these are files

10:36I mentioned S3 or blob store, like it

10:38This seems to be a database of some kind, but is that what Iceberg is? Is Iceberg like a server process? Do I go run it somewhere? Can I go buy it at iceberg.te or, you know, wherever it is? The answer is no

10:53It really is a specification, and all of these things are very pluggable

10:57Like this catalog can be the Hive meta store

10:58It could be a JDBC database

11:00It could be SQLite running on Raspberry Pi, if you're into that kind of thing

11:04So it's just a database that is keeping track of, uh, the rest of where the the Metadata files exist

11:12In the world of streaming, Iceberg now lets us store our data in a data lake, have very much, you know, relational database-like semantics around that data, even though it is really Parquet files in a database, and gives us flexibility into how we're going to query that

11:31Now, uh, there isn't a server, there isn't some like main way in

11:34There's a set of libraries that you can use

11:36You can write Java code, Python code, uh, Flink code, Spark, whatever, you know

11:41Then there's other tools like Presto that'll even let you do sophisticated joins across tables in here

11:47There's all kind of Tools on top of Iceberg that obey this open standard, uh, that let you do the kinds of querying that you want

11:53And it opens things up in the world of streaming that are really exciting

11:58So a thing that we at Confluent do with this is, we realize that, you know, uh, most of the time what feeds the data lake, oh, it might be a batch process, right? But, you know, these days I think we know it's probably a Kafka topic, or more than one Kafka topic, that's, that's feeding that data lake or writing to those parquet files

12:21And why have this before and after thing where I've got this streaming world over here and I'm going to do some Minimal stream processing, dump it into the data lake, and then Iceberg is happening over here

12:33Why not just, you know, with an option, enable Iceberg semantics on top of that data? And that is what Confluent calls Tableflow, which is a way of making the data in a topic accessible as an Iceberg table

12:48So changes to that topic, changes to the schema through the schema registry, and the governance features of Confluent Cloud, keep all of the appropriate manifest files, lists, and metadata files updated

13:03So, however it is you are accessing Iceberg, you point it at your Kafka topic and that works without the

13:08The old copy from there to here, kind of thing, which is kind of cool

13:13A lot of neat things are happening

13:14Obviously, there is so much more to say about Iceberg, uh, how it deals with updates and upserts and deletes, and uh, how it pulls off transactions in terms of, you know, the steps it goes through

13:27So much to say, so much more to learn

13:28I hope this gives you a good overview of the reasons, the logical architecture, and the modern implications of an important technology like Iceberg

13:38Thanks

13:43[Music]