00:13Hey everybody, this is Alex Merced, developer advocate at Dremio

00:27Recently, on my LinkedIn blog, I published this article, "Creating a Local Data Lakehouse Using Spark, MinIO, Dremio, Nessie." So what I'm going to do is, I'm going to walk through the steps of that article in this video, showing you sort of exactly how to execute this lakehouse on your laptop, okay? So bottom line, and again, the point of this is not that you're going to run a full-on data lakehouse on your laptop, but you may want to explore the whole concept of a data lakehouse and see what it's like to use things like Dremio, MinIO, Nessie, Apache Iceberg, and This will kind of give you a setup that you can set up locally on your laptop to evaluate these things, um, to experiment, to play, to understand better how you would implement this into your data infrastructure

00:41But with that, what I have is, I have Visual Studio Code open, okay? And I do have Docker installed, so if you don't have Docker installed, you will need Docker

00:48I think everything other than Visual Studio, Visual Studio Code, and Docker, you should be good to go, okay? But basically, what I've done is, I've opened up an empty folder here in Visual Studio Code, okay? And what we need to do is, we're going to use Docker to

00:58Create an environment with older pieces

01:09Okay, so essentially, we're going to be having a container that's going to be running Apache Spark and Jupiter Notebook, so that way we have the ability to do like, uh, PySpark, um, and actually, you know, uh, to do sort of streaming and batch ingestion

01:20Uh, we are going to have a Nessie catalog running in a different container, and that'll be our catalog that tracks all of our Apache Iceberg data sets that we can then connect to Spark or Dremio

01:33So basically, I can ingest data from Spark and then immediately read that from Dremio, uh, without really not much of a hiccup at all, okay? And, uh, Dremio will be our query engine, okay? And, uh, you know, basically it allows us to query our data, accelerate, uh, queries on those data sets, and do a lot of really cool fun stuff, okay? So those are gonna be your tools

01:56Again, you can read the article and follow these links to learn more about any of these specific things, but let's just show you the setup, okay? So essentially what we're going to do is we're going to use Docker Compose

02:01So Docker Compose allows you to create a file called Docker Compose dot YAML, and in this file, what you can do is you can define several different containers you want to set up to create an

02:09environment that's all on a shared network, okay? I already pretty much have it all kind of sketched out for you here, so I'll just literally just copy and paste what we have from the article, okay? And essentially what we're doing is we're creating several services, which are just basically the containers that we're starting up

02:27We're gonna have our Dremio container, okay? We're going to have our MinIO container, which is going to run MinIO, okay, which is going to be where we store our data

02:33It's basically an S3 compatible uh storage layer

02:48We're gonna have our spark notebook is using a container that I made that

03:02Has Spark 3.3, uh, and Jupiter Notebook running, so that way you can run, uh, Spark jobs against that, uh, instance, and then our Nessie server, and they're all gonna be running on this shared network called Iceberg environment, okay? So basically, once you have a Docker Compose at YAML that defines all the different services that you plan on running on the shared network, what we can do is we can, uh, run these containers, we can do one by one, okay? But also, we're going to want to make sure that we have certain environmental variables that particularly our Spark container is going to depend on, okay? So you see here we're

03:13Going to be using this .env file, so I need to create that file

03:23So I'm going to create a new file called .env, generally see sort of standard file you would use for hiding sort of your, your sensitive details, okay, which details you're going to need for this exercise, I put in here, okay, so we can just copy and paste that

03:34Most of these are going to be fixed, okay, so such as this region

03:40Um, these we're going to be getting in a second, so you'll see like right now they're not filled in because we're not actually using AWS, we're using MinIO, okay, which again is compatible with S3, so it follows the same sort of rules, the same

03:46APIs, um, but we'll need to get, once we start up the MinIO container, we'll need to get our access key and secret key, uh, to access that MinIO server

03:54This is where our MinIO Server will live

04:02Um, basically, we'll have a bucket called Warehouse, which we'll create in a moment, and here's going to be our Nessie API URL

04:08Okay, so there we go

04:15So what we're going to do first is we're going to run the MinIO server, so I'm going to do is I'm going to open up terminal here in VS Code, and overall, I'm going to end up needing four terminals, essentially one for each service we're going to run

04:21You could just do Docker Compose up, so if you have Docker, Docker installed

04:28If I were to do Docker compose up, it just reads the Docker compose file and then just starts up all the different services, but then you have all the logging for those services in one terminal, and that can be kind of hard to track

04:39So what we're going to do is, we want to run one at a time

04:51So first, we're going to say, "Hey, I only want to start the MinIO server." Okay, so Docker compose up the MinIO server, and that's going to start up that MinIO server, okay? And if we go back to our Docker compose here, we can see that the MinIO server is going to be exposed to us on ports 9000 and 9001

04:57Okay, so I can now head over to localhost

05:03Localhost always refers to my computer, and that's going to be a difference for you

05:09Which one's the, I think it's 9000

05:13One is the web application

05:18Yeah, nope, nope, that's, I want to go to 9000 then

05:21Oh, that's right, spot

05:24Okay, and then again here in our Docker compose, we define what the username and password is going to be, Mino admin

05:27So this is menu admin, menu admin

05:30Okay, and I click login, and that's going to log me into the menu, um, dashboard

05:34So there's two things I want to do here

05:37One, I want to create that warehouse bucket, so I'm going to go in here and just create a bucket

05:41We'll just Call it Warehouse, so that way it follows the environmental variable we created

05:45So I create that bucket, that bucket now exists, and then we're going to want to grab, grab an access key

05:51So I'm going to click here, say create access key, and I'm going to copy these credentials over to that dot EnV file

06:04Okay, over to that EnV file, okay, and then I'm gonna copy over the secret access key

06:10Okay, awesome

06:15Okay, and that doesn't, doesn't look like I copied that over right, so let me just try that again

06:19Ctrl C and put that there, Ctrl V, there we go

06:24Okay, cool, so those two are there, and then let me just make sure it's created, so click

06:31Create, and I don't need to really download it, I'm good

06:35Okay, I have that access key saved, okay, so that's all fine and good

06:39Okay, so we're kind of done with MinIO, I'll leave that tab open

06:42Okay, so let's head back to the blog

06:45So now that I have that filled out, I'm good to kind of open up, uh, the rest

06:49So first thing I'm gonna do is I'm going to open up a new terminal, so I'm going to click here, this little plus button, which will open up another terminal, so see now I have two terminals open, and this one we're gonna start up Nessie, so Docker compose, uh, Nessie, and that's going to start off the Nessie service, uh, Nessie by default

07:07Because we just use pretty much the default settings, as you see here in the Docker Compose, okay, nothing too fancy there

07:13So basically, it's going to be using memory

07:19It's basically the default stores memory

07:25You can configure kind of any back end, like a RocksDB or a MongoDB, that kind of store all the metadata that Nessie uh tracks, but we're just using the memory story just because we're just kind of using sort of this lightweight evaluation type environment

07:31So it's basically saving all that data in the memory of the container

07:38Um, and then two, um, basically the default authentication is none, so there's no

07:46Authentication available

07:53You do have two other options with Nessie, which is token-based authentication, which is what Dremio uses with Dremio Arctic, and then there's also AWS credential authentication, but you can configure, uh, if you were to set up your own SC server, so you have lots of options

07:57It's a very flexible tool

08:02Okay, now basically that's good to go, that's running

08:05Okay, nothing we need to really do there

08:08So now I'm going to hit plus to open up another window, another terminal, and now we're going to start up the Spark notebook

08:11Okay, so I'm going to hit Docker Compose up Spark notebook

08:16Okay, awesome

08:22Okay

08:27So that's starting up the Spark notebook container

08:30Now, when that's done, it's going to give us some output that we're going to want to grab

08:34So you see here, there's this output right here that tells us where it's running

08:37The reason we need this is because there's a token in the URL that we're going to need to get access to it

08:43So I'm just going to literally just copy that over, and then I'm gonna open up another tab here in my Chrome browser, and here we go

08:48Okay, so see here, I have Spark running

08:54Okay, I mean, I have Jupiter notebook running, and then I can run uh Spark jobs against it because Spark is running in this

08:59container, um, so it's all good

09:03Okay, cool

09:07Now, last thing I want to do is, uh, spin up Dremio

09:10So open up one more terminal, Docker, compose up Dremio, and I'm gonna let that set up

09:19That'll take a bit, so we can just continue on while it does that

09:27So first, we want to do is we want to ingest data into our table, so we want to actually create an Apache Iceberg table, um, and whatnot

09:30Okay, so Dremio is running

09:34Okay, so here I already have the script, so you can kind of test it

09:37Again, you can always kind of do other stuff, but we'll copy this code over and we'll walk through it

09:40So what I'm going to do here is in my notebook, I'm going to create a new

09:47notebook by clicking on new over here, okay? Create a new Python 3 notebook, okay? That's going to open up a new notebook, okay, and there it is

09:59And then I can just, you know, copy the code from the blog, okay? Give that a second to kind of figure itself out

10:07So basically, what I'm doing here is I'm grabbing all those environmental variables that we defined in our .env file, because they should be available to this container

10:14How do we know that is because we said that this particular container is going to use that environmental file to define environmental variables, okay? And actually, we really don't need these

10:26Prints, I just left those there

10:32Okay, but essentially here we're configuring our catalog

10:38Okay, when it comes to working with, so essentially, just to go through each of these settings, so that way, if you were to configure different catalogs, you kind of understand the vibe here

10:44Essentially, in Spark, what you're doing is, first, we're downloading any kind of jars that we may need, so we're not going to assume that, hey, the Apache Iceberg jar and some of these other jars are inside, they're not in this particular container

10:55Okay, so in this case right here, we can specify, hey, any containers that Spark should

11:01Download when it starts up, so in this case, we're using Iceberg 1.1.31, we're using Nessie version 67

11:11Um, we're using this version of the Amazon AWS SDK bundle, that's for working with S3, uh, type inputs and outputs, um, and the URL connection client for AWS SDK

11:18So all those will kind of allow us to kind of function in the type of workflows that we're creating here, which is using Nessie as our catalog, Apache Iceberg as our table format, um, you know, with that S3 storage layer, with that S3 compatible storage layer

11:27Okay, now what happens then? You configure the SQL extension, this is going to allow us to use all the

11:37Special SQL for like Nessie and Apache Iceberg, so we have the Nessie Spark extensions and the Iceberg Spark extensions right there

11:42And then here, we configure the actual catalog, so notice we give the catalog a name

11:48I give it the name of Nessie, it doesn't have to be called Nessie, no, you can call it whatever you want

11:53That part of the property name is "stay off the match" because you're configuring that same catalog, but I.E., as I'm creating a catalog called Nessie and these are the settings for that Nessie catalog that I'm using

12:04Uh, so you can see here, I'm using the Nessie catalog implementation, this particular

12:11Nessie catalog that I have running in that separate container does not have authentication, because I'm using the default, which is none

12:17Um, you know, you would change the token if you were using token and have another property called token

12:22For AWS, it would just go about based off your AWS credentials, and then Nessie allows you to do branching

12:29Okay, so basically one of the great things about Nessie is that not only does it allow you to catalog your Iceberg tables, but it provides catalog level versioning, so not only is every time you make a change to any of your tables does it create a commit of the Catalog, but you can also create branches, so you can isolate changes on your catalog to create like zero copy, uh, zero, zero copy clones, so that we can do experimentation, um, which also enables things like catalog level rollbacks, catalog level tagging, etcetera

12:57But generally, this setting will allow me to specify what's the default branch, so I'm going to say, "Hey, I'm working on the main branch," which is the default branch right off the bat

13:03But let's say I did have other branches that I use for other purposes, or I have a certain branch that I want to work on from the get-go that I've created in the past, I can set that there

13:10Now, since we are not using AWS, we're using MinIO, we have to tell Spark to kind of use a different URL endpoint when it's making these S3 API calls, to not send them to AWS, but to send them to our MinIO server, so that's what we're doing here

13:26So we're taking that variable from our .env file, this is where it's, the warehouse is where it's going to write the data when it saves the data, which is going to be, uh, again, going to our .env file in that bucket called Warehouse

13:40We're using S3 input output, so that's, that's the classes it's going to use when it writes files, and those that, hey, I'm running the S3, not to, uh, HDFS or some

13:47other file system, and then these are my AWS credentials

13:55They're also inside the environmental variables, so one of the two we're going to end up allowing uh, Spark to kind of access things

14:02Okay, then this is just something to kind of confirm this park is running, just a print there

14:08Then here, we're creating an Iceberg table, so you just, it's a create table save and creating a simple table with one field, string name, which is a string, and you see here in Spark, we have to use this using Iceberg language to specify that, hey, for this table, we're using the Iceberg libraries, because it is very possible that you might be Doing Parquet tables, you might be doing Delta Lake tables, Hoodie tables, all within the same Spark session, which means, you know, you'd have all those configurations all here when you're configuring all these catalogs, but, um, you know, you have to specify what you're doing

14:19So this should create the table

14:31Okay, this is going to insert into the table

14:39Okay, so we're going to insert three names

14:48Okay, so there's my name and like the names of my colleagues, Depend Camara, Zoomdar, and Jason Hughes, and then we'll query that, so you can see that, hey, I can query from Spark as well, and again, this could be anything, this could be

14:56Me ingesting a stream through using Spark Streaming, this could be me doing batch streaming and scheduling this, that occur on a certain interval, and I could be ingesting it from, uh, anywhere

15:02Okay, so it doesn't necessarily just be arbitrary, hey, inserts, I could be grabbing data from a parquet file, uh, I could be grabbing data from a database, you know, you can design whatever you can imagine, but the main thing is that I can then load that data into an Iceberg table, but again, I can create the Iceberg table and then, and if I won't need to do that, the table already existed in the catalog, and then I can just insert the

15:28Data as needed, or use whatever the proper SQL is for your particular use case

15:34So I'm gonna just put my cursor all the way here at the top and hit run, and this is going to start running the spark drop

15:40So first thing you're going to do is it's going to start downloading all the libraries, so this will take a second

15:44Okay, that didn't look like it started running, so let's try that again

15:49Oh, there it goes

15:53Okay, so right now it's downloading all the libraries and configuring the spark session, making all those catalog configurations, and getting all that stuff

16:07Okay, you can see it's a little overwhelming when you see all this

16:21Logging output, but if you read through it, really it's just downloading the libraries and then starting the Spark session based on our configurations right here, and then it's just following our instructions that we've laid out for it, okay? So that's gonna take a sec, okay? So while that does that, let's go head over to Dremio, so that's going to be on localhost, should be 9047, I think, yeah, okay? So when you start that Dremio container, okay, what's gonna happen? It's going to ask you for your name, so you just give it a name and a username, and oops, not I understand, it doesn't really matter, and then Alex Merced at

16:35Dremio.com, and then we'll just put in the password

16:44I'll just literally put "password" for now

16:54Okay, oh, once we have, once a longer password, okay, so we'll just do "password one," "password one." Okay, so here we are on the Dremio dashboard

17:03Okay, so when you're on the Dremio dashboard, if it's essentially what happens is you have sources, so these are your actual, like, hey, where does the data exist? Okay, and we can add sources, and there's lots of different possible sources we can connect, as you can see here

17:13Okay, so we can kind of coalesce all of our data in one place

17:19Now, we're going to be using our Nessie catalog, so we'll be Showing you how to set that up in a moment, okay? But then once we have our data collected, we can then, you know, join that data, create views on that data, and then those views can be organized into spaces, okay? In those base, and then the spaces bases can have a subfolder, uh, and then, and then data sets inside of them, and all of that can then be controlled as far as like who can access it, what columns they can access, what rows they can access, you have really granular controls

17:48And not only that, let's say we add a space, so we'll just say, you know, uh, data product one, okay? So imagine like each space is one data product, okay? I can

17:58Control this data product, so again, if you're using like the cloud version or the Enterprise version, I'd be able to click on the settings here and be able to kind of set controls for it

18:04Okay, the, uh, right now we're using the docker container version of it, um, but then I can go in here, I can create additional folders, you know, we'll just say, hey, you know, let's say you're doing like a medallion type structure, so you know, bronze, gold, or silver, gold, silver, gold

18:39Okay, and the cool thing is, if I click on this, okay, you see like this is a little panel up here that can pull out, and there's a Wiki attached to that folder, okay

18:46There's also a Wiki at the space

18:52If I go here to the space, there's a Wiki at the space level, too, and that can be edited using markdown, so that we have documentation for your data sets

18:58Okay, so you have a lot of flexibility and, like again, how you organize your data sets

19:02So let's go back and check it out, Spark, see if Spark is done

19:07Okay, so we ran all that, okay, and there's the jobs, and it successfully created the table

19:11We can see because we were able to successfully query the table

19:15So if that is the case, then I should be able to head over to MinIO, go back to our object browser

19:20If I go inside Warehouse, I should find the data and There it is

19:24Okay, so we see there's a folder for our new table, and generally when you see an Iceberg table, you're going to see that inside that folder for that table, there's two folders: a data folder where the actual parquet data files are saved, and a metadata folder where all the metadata files are saved

19:36So you see here, like here's a metadata.json, and notice there's two of them

19:42Okay, one of them is going to represent the creation of the table, and the second one is going to represent the transaction when we inserted data into the table, anytime you change the table, whether you're changing like the table

19:52Schema, you're adding data, deleting data, creating a table, you're going to create one of these metadata.json files, um, and generally that's what the catalog tracks

19:58The catalog says, "Hey, which one is the newest one?" So then an engine like Dremio can then go over to Nessie and say, "Hey, what is the newest metadata.json for this particular table?" And then that, from there, you can kind of figure out and construct the table

20:18Okay, if I go back over here to Warehouse table and then data, here we have the parquet files with the actual data

20:24Okay, I didn't partition it, didn't kind of put any kind of partition rules, so gently

20:30It's basically treating each record as a partition in this particular right, so I have three parquet files, each with one record

20:34Okay, partition your data, good practices

20:39Okay, but um, yes, so that, so we know that the data is there, wonderful

20:43So now all we have to do is we have just hit the connect Dremio to a Nessie

20:48Okay, so what I'm going to do is I'm gonna put Dremio here on the side, I'm gonna go put the blog post on the side

20:59Okay, there we go, and we can probably give Jeremy a little bit more space here

21:04Okay, and let's kind of head over, so we can see that we, we've done all that, now we're setting up Nessie, so I'm gonna click on ADD

21:10Source here in Dremio, okay? And I am going to first put the name and just give this source a name, so just call it Nessie

21:23Then we're going to put the Nessie endpoint URL, okay, which is going to just be HTTP slash Nessie

21:30Now, why is it Nessie? That's because, just to kind of explain the why, when you create a Docker Compose file, all of these are on a shared network, and when they're communicating with each other, they all basically can use the name of that service, um, that's kind of like their domain name, okay? So since this is called Nessie, and I also reinforce it by giving it a container name of Nessie, I'm able to do like HTTP, Nessie, and any of the other containers should be able to actually, like, communicate with that specific container from that

21:58Okay, so it's just, that's just an aspect of how Docker works

22:04Okay, so back to Dremio, HTTP, Nessie, um, okay, that's gonna be on port 19120, okay, slash API slash V2, okay, and now keep in mind, notice this is different than what I did in Spark, okay, because right now the Nessie Spark Library still uses the V1 version of Nessie's API, while Dremio's Nessie connector uses the new V2 version of the API

22:24They do functionally the same thing, just slightly different endpoints, um, so just make sure that

22:31You're using the right suffix here when you're connecting, when you're setting it up in Dremio, it's going to want the V2 endpoint

22:38Um, again, then you choose the authentication, none

22:42Okay, so that gives you the information needed for connecting to Nessie

22:46Okay, but Nessie only stores the reference to the metadata

22:50Now, the things that Dremio is going to also need access to, where the data is stored

22:56Okay, so generally, so this first panel gives us access to the catalog, and then we also have to provide Dremio how to access the data where it is stored once it gets that reference from Nessie, so we're, you know

23:10Theoretically, using MinIO, which is again AWS compliant, S3 compliant storage layer, so essentially here we're just putting in our information from MinIO, so I'll just go back to that ENV file and grab my access key

23:29Okay, I'm gonna grab that access key, I'm gonna go grab that secret key, okay, and generally, you know, I wouldn't expose these things, but again, it's just, it's not a real AWS access key, this is just, you know, a Docker container running on my computer that will be destroyed after I'm done recording this video

23:47Um, okay, then the AWS root path, okay, so we can just say, okay, I'm gonna save my, my warehouse bucket

23:52Will be where I want to display

23:56Okay, that's where I want the kind of things to kind of work out of

24:00Um, again, this is if you use the access key

24:03I could also assign an IM role that has the right authority

24:06That's if you're using AWS

24:09For many of this should work just fine

24:12Now here's the thing, Dremio is not going to know automatically that, oh, you're using MinIO, we should go make an API call somewhere else

24:16You have to tell it that, in the same way we have to tell Spark

24:20So the way we do that in Dremio is we're going to use this add property, uh, feature here, and we have to add a few properties

24:24Okay, the first one

24:30is FS or file system, s3a, so we're talking about S3, S3 API, the path style access, okay, so it's just basically saying what kind of access style to the S for the S3 API, we're just going to set that to true, okay, so let's go set it up into kind of the style that MinIO is going to need, okay, but we also have to tell where to send those API calls, so FS dot um s3a dot endpoint, okay, and that's just going to be our, our uh, MinIO URL, okay

25:10Now, you don't include the HTTP or HTTPS when you do this, so this would just be MinIO:9000, okay, and again, why is it called MinIO? Because if I go back to my Docker Compose file, okay, can we take a look? At here, I could have used MinIO server, or I could have used the container name, MinIO

25:20Okay, so it has that name, cool

25:30And then I'm going to add property again, and then we have to do is basically set it into compatibility mode

25:41This is a mode in Dremio that allows you to use S3 compatible storage layers

25:46So at dremio.s3.compat, that just tells Dremio to be in that compatibility mode

25:53Okay, and now again, I'm doing this on Docker on my local machine, so these connections are not SSL encrypted, so in that case, I want to turn off the encrypted connection, and then that should be all I need to do

26:06Okay, so now we've basically Said, "Hey, here's how you access the catalog," which was pretty straightforward, and here's how you access the storage layer, okay, wherever the data is stored, that the catalog will point you to

26:17And there's some advanced options for like caching and other stuff if you want to play around with that, but I hit save, okay? And that's just going to take a second

26:22It's going to go first, connect, you know, hit the catalog, make sure that it can access the catalog, and make sure you can access the storage, and make sure that everything is kind of A-Okay, and there's things wrong with the settings we provided, okay? And then Eventually, we'll be able to start querying our data, okay? But you can see here all, basically, once this is like loading, we're going to be able to see our tables that we created in Spark, okay? And basic bottom line is there, oh, the catalog becomes the Arbiter of Truth

26:52Nessie becomes the Arbiter of Truth between Dremio and Spark, so as data gets just in Spark, it becomes immediately visible to Dremio, okay? So I probably missed some steps, so you see I have this error here, okay? And then one of the nice things about having like the Docker container spinning up in multiple uh terminals is that now if I really want to give more

27:16details on this error, I can go head over to VS Code, open up the container, open up the terminal that I have Dremio running in, to go inspect the logs to kind of see where that mistake happened

27:22Okay, the president we calculated does not match the signature you provided

27:28Okay, so it sounds like I copied over the keys wrong, so let me go recopy that over

27:35So, um, I'm gonna play with that

27:42Okay, so let me head over to here and I'll be right back

27:47Yep, and that was exactly it, I was missing one character off my secret key

27:54So, um, that's now set up and see like now I can see uh the Nessie Source, okay, and I can see the table that we Created in Spark, okay

28:02Now, just to show you a couple of things, the cool thing about Nessie is, again, you can have branches

28:07So, if I had created branches from Spark, I'd be actually able to see the branches here, okay? So, I can see those branches here, and I can switch between them and actually take a look at the different commits that are done through there

28:13I can also create branches from Dremio, but we'll come back to that

28:19So, I can go here and say, "Hey, I want to look at the names data set that's inside of Nessie," okay? And I can just run, and again, right now this is gonna be using my laptop's computing power, so it's not going to be

28:31The same uh processing performance that you're gonna necessarily see when you actually have a full-on, you know, production a Dremio cluster, same thing goes for Spark, but if I hit run, it's going to run that query, so it'll be pretty quick, and basically, we'll see that data from that query in just a moment

28:51So basically, um, showing up, showing up, there it is, and then you can see like the data come in, and again, it goes through that whole process of then hitting that Nessie catalog saying, "Hey, where is the latest metadata file for this particular table?" Um, then basically goes to MinIO, grabs that metadata.json, is able To reconstruct the table from all those data files and then queries it, okay? If you've seen any of my Iceberg architecture videos, um, you'll probably be familiar with that process, but you see I can query it, okay? I made it in Spark, created it in Dremio, and the cool thing here is I can, you know, make a view, okay? So let's say, hey, I only want to say, select, um, limit one, let's make it simple, limit one, okay? So we only want the first person, okay? Now, if I run that query, okay, there it is, just the first person, and see that was pretty quick

29:40Now, what if, what I can do is I can save that as a view, so I can click here, I could save it as a script, if I just want to rerun the SQL later, I don't want this to be like a data set that people can access, but I can click save view, and then I can save that logical view under a space

29:51Okay, so we can say, "Hey, this is a gold data set," as an example

29:58So it's just, just Alex, we'll call it, okay? And now what I can do is, if I go back here and I click on data products under gold, see, there's just Alex, and see, now there's this curated view of the data

30:17So this is how Dremio kind of enables that sort of no copy architecture, because I can connect all my data sets, join them, create alternate versions of them through SQL statements, save them as Logical views, and basically from the user standpoint, when they log into their Dremio account, they can just see, okay, hey, there's this data product, this data product

30:30They can see the data sets the way they need to see them, with the column names they need, with all that sort of last mile ETL work already done for them, okay? And it just makes it a really easy way for the end user to interact with the data, so the data engineer would kind of, you know, connect all the data sources, curate these spaces, and then that data consumer would then be given a data analyst, data scientist, would just be given an account

30:54Where they can log in, they'll see all the data organized, see all the documentation in the wiki, and just be able to access their data sets, not just through here in the UI, but they can also access it through a REST API, so that way they can use it in PySpark

31:06They can access it through ODBC, through AeroFlight, so that way you can bring it into like another book or a BI dashboard, and there's integrations with things like Power BI, a Tableau right here in Dremio

31:20So this makes it really, really easy for the end user to access the data, and again, we can use Spark for ingestion, and the Nessie catalog acts as sort of the glue

31:31To make sure that that data is discoverable across all of our tools that support that Nessie catalog, okay? And again, I'll just show you like a quick create rank statement

31:37Let's see here, create branch

31:43[Music] Okay, uh, we'll call it ETL, okay? And if I do that, and again, I'm doing it in the context of the Nessie, so if I hit, and I think I actually have to specify, I think it's in Nessie, so that way, notice that I'm doing it in that specific thing

32:01If I hit run, may not remember the exact syntax I have my head

32:06Yep, that worked, okay? And see, now if I click over here, I have two branches, okay? And again, that branch is accessible in Spark

32:13And accessible, and Dremio, it's not a thing that's created in Dremio, it's a thing that exists in the Nessie catalog

32:18So now what I can do is, I can actually go, you know, say, "Hey, I'm working out of this branch now." So if I go here and I switch this over to main, to the ETL branch as my contacts, okay, I can then, like, create another table, create table, um, create table names, too, name string, name varchar, okay? And here, you don't have to use using Iceberg, I can just create the table, and let me actually just insert some records

32:39That's another cool thing you can do, driving

33:00You can run multiple SQL statements at a time, so I can be like Insert into names two, and actually, it should be messy.names too

33:21Names to values, and then we just again put Alex Merced, just to kind of get straight to the point, and then I can hit run

33:27It's going to run these SQL statements

33:34We'll see you created a table

33:40Now it's inserting into the table, and that's done

33:46Okay, and if I go back over here, take a look at like the state of my Nessie catalog, see, I see names and names two, but if I go back to the main branch, I only see names

33:52Okay, so you see, like, I isolated the existence of that table into that branch, um, and then again, if I made any changes to names on the branch

33:58That would have been isolated too, so you have that uh, that nice sort of uh, form, and again, you have access to all of that in Spark too

34:04There's a bunch of SQL, SQL extensions give you all that sort of branching and merging flexibility over there in Spark

34:11So, um, hopefully you guys enjoyed that

34:15My name is Alex Merced

34:18Now again, if you want to bring all that down, so that way you know how to turn all this off, I'm just going to create another terminal

34:22Instead of doing each one one by one, I'm just gonna do Docker compose down, and that's gonna just shut everything off, okay, and that'll be shut off and good to go, okay, so I'll see

34:36You later, hope you guys enjoyed this

34:36Make sure you subscribe, like, leave a comment

34:37I'll see you all later.