00:00In this video, we're building an advanced AI agent in Python using Langraph
00:04Now, this isn't going to be a basic chatbot
00:06This is a multi-step deep research agent that will pull live data from sources like Google, Bing, and Reddit
00:13Now, this tutorial is not for beginners
00:15I'm going to be covering advanced Python concepts, complex architecture, and best practices for building agents that go far beyond just a single prompt response
00:24Now, for this video, like I said, we're using Langraph, Python, and I will also be using bright data to access real time web data
00:32Either way, even if you don't want to build a web-based agent, you can still learn a lot from this video because I'm going to build really a pretty complex agent and show you how to structure that in Python
00:42So, by the end, you're going to have a fully functioning project and the skills to build powerful production AI systems
00:50So, with that said, let's dive in and let me show you a quick demo of the project
00:54So I'm in PyCharm here and I'm just going to give you a quick demo of what the finished project will look like
00:59Keep in mind that you can adjust this a ton and this is really meant to give you a solid base
01:02Anyways, let's have a look at what we can do
01:04So as you can read here, this is a multissource research assistant, but you can use it for a lot of different tasks
01:09It says ask me anything
01:11So I said, should I move to Dubai from Canada? And then what we do is we start in parallel searching both Google, Bing, and Reddit
01:17Okay, so we can actually do this at scale and search for like thousands of different results
01:23So you can see we searched Bing for this, we searched Google for this, we searched Reddit for this, and then what we start doing is waiting for the results
01:30So essentially what Bright Data is going to be doing is it's going to be pulling all of these results into a large snapshot for us
01:34See, we get all the results here
01:37We download it and then we're able to actually, for example, parse through all of the Reddit posts
01:42In this case, I just did 75 of them
01:44We analyze the Reddit post
01:44We find the titles that actually make sense for our search query
01:47We then go, we find those Reddit uh posts, sorry
01:50We download all of the comments from those two Reddit posts
01:54You can see right here it says it downloaded 26 Reddit comments
01:58We then take all of that data and we analyze it
02:00So we analyze the Google results, the Bing results, and the Reddit discussions
02:03And we synthesize all of that data into one final answer
02:07And then it gives us the response right here
02:09So it says, you know, taxfree income, cost of living, lifestyle, and environment
02:12And if we scroll over here, it tells us where it actually got this data from
02:16So from Google and old assets
02:18Here it found this from Reddit r/duby and it gives us kind of all of these sources and quotes where it found this information from
02:27Same thing I asked it should I buy AMD or Nvidia stock? Same process
02:29It goes through finds all of this relevant web data finds all of these different posts and then as we scroll down you can see here in this case there was a lot more comments that it analyzed and then it gave us this comprehensive response breaking down whether we should invest in Nvidia or AMD
02:42And ultimately it gives us a conclusion here and says Nvidia appears to be a more straightforward choice for those prioritizing immediate and robust growth in AI who favor it strong financial and marketing position
02:53Okay, so there you go
02:54And then I just stopped the agent here which is why we got that kind of interrupt message
02:57So overall this is a very powerful agent and the interesting thing about this is the way that we're going to write this is it's very scalable
03:04So I could run this agent thousands of times in parallel
03:06I could scrape thousands of different posts and nothing really changes here
03:10You'll see how we do that later on
03:12But because of the technology that we're using and the way that we write this, we really can run this actually in production in a scalable format and it's going to be pretty fast to actually execute
03:21It doesn't take a very long time due to the services that we're using
03:24Anyways, with that said, I want to quickly talk a little bit more about search because that's a really important part of what we're doing here
03:30And then we'll get back onto the computer and start coding all of this out
03:33So, now that we've looked at a quick demo, I want to briefly touch the current problems that exist with search because that's what this agent is going to do
03:40it's going to be searching the web
03:41Now, most AI agent systems today share a pretty major limitation
03:46They can't access the full range of data that's actually relevant to the problem
03:50So, relying on a basic search API, or just a single plug-in means that you're usually only seeing a fraction of what's actually available on the public web
03:58So, important sources like live social media sentiment, real time crawling, or historical trend data often go untapped
04:06Now without them the results are incomplete and the decisions these agents make can be based on outdated information and opportunities are going to be easily missed
04:14Now on top of that many setups require stitching together multiple APIs that don't work smoothly together and they leave the agent with a narrow or fragmented view of the entire world kind of based on the web
04:27Now with that said that's why for this video I mentioned we're going to be using Bright Data
04:30Now Bright Data has been a long-term sponsor of this channel
04:34I've worked with them on many videos now and what they have here is a web discovery API that provides a much easier way to access a wide range of public data
04:42So rather than just basic search results, it can pull all kinds of information like live SER data from engines like Google, Bing, and other search engines
04:51Realtime sentiment from platforms like Twitter, Reddit, and Tik Tok
04:56In this video, we're going to be scraping Reddit, for example
04:58They have historical web data that goes back years
05:01and then insights from answer engine such as perplexity, Gemini, chat, GBT, etc
05:06Now, that's because Bright Data handles the crawling, parsing, and does this reliably in one unified API so that we can focus on building the agent rather than trying to set up these really complex web scrapers
05:18Again, I've used sprite data many, many times in the past
05:22Essentially, just a smarter, easier way to be able to scrape the web
05:26And we're going to integrate that tool into our agent here in this multi-step kind of orchestrated flow so that our agent essentially grabs all of this real web data based on what we're asking for, analyzes it using models like chat GBT, and then gives us a really competent response that's kind of followed the strict protocol that we've set up
05:43So with that said, let's get onto the computer here
05:47Let me start explaining the architecture of our agent
05:49and we're going to start kind of scaffolding this out, building it, and then going step by step because I won't lie to you, there is a lot of code here
05:56And also, I'll quickly mention that all of the code for this video will be available from the link in the description
06:01Anyways, let's dive in
06:03So, we're back on the computer now and we're going to start coding this out
06:06Now, first I want to quickly explain the Langraph architecture that we're going to use for this agent just so you can understand what it is that we're about to build
06:13And with that said, if you're unfamiliar with Langraph, I do suggest you have some background with that before following along in this video
06:20Otherwise, it might be a little bit confusing
06:21So, I'm going to put a video on screen right now that explains Langraph in depth that you can follow along with to get some context before going into more of a complex project
06:30Okay
06:30So, anyways, we're going to be using Langraph and essentially what's going to happen is the following
06:36The user is going to ask some kind of question and we're going to simultaneously in parallel go and search Google, Bing, and Reddit for information based on their question
06:44Now, in theory, we could search a lot more sources as well, but I'm just keeping it a little bit slim for this video, so it doesn't take us 25 hours to code this out
06:50Okay
06:54Now, after that, what we're going to do is we're going to wait to analyze the Reddit posts
06:57The reason for this is that the Google results and the Bing results are extremely fast because Bright Data already has them indexed
07:03So, we get them in like a few seconds
07:05Whereas the Reddit post can take a little bit longer because we're doing manual web scraping
07:08Well, Bright Data is doing that for us and pulling all of the relevant posts
07:12So, in this case, we have to wait for the Reddit post
07:14Takes a second
07:15And then once we get the Reddit post, what we're going to do is retrieve all of the posts that are actually related to the prompt that we passed in
07:24So, we're going to search Reddit
07:26It's going to return a bunch of different results for us
07:27Then we're actually going to analyze those Reddit posts
07:31We're going to pull out the ones that make sense
07:33And then from those posts, we're going to retrieve all of the comments on those particular posts
07:39Hopefully that makes sense
07:39But that's kind of these two steps here
07:40Now, after that happens, we're then going to analyze all of the results that we got
07:46So, we're going to analyze the Google results, the Bing results, and the Reddit results
07:49This is so that we cut down the amount of information that we have before we synthesize that all together into one larger prompt
07:54So we have these three kind of smaller prompts that are focused on pulling information that we need from each source
08:00Then we take all of that and we pass that to kind of a synthesizer which is then going to take all three results from here and synthesize that into one final answer which we'll end up getting here
08:12Okay
08:12So this is kind of the architecture or the graph that we're going to build
08:15And of course, when I say lang graph, that's referencing this graph, right? Like we're building this graph essentially where we're flowing data through this and eventually getting this final answer
08:25Okay, so that's what we're going to build
08:28So what I'm going to do now is go over to PyCharm and start setting this up
08:30Now, for this video, you can use any IDE that you want, but I do typically recommend PyCharm for larger Python projects, especially when you're working with AI or modules like Langraph, Langchain, etc
08:42And I do actually have a long-term partnership with PyCharm and you can check them out and start using it for free from the link in the description
08:48I always recommend at least try it
08:50If you don't like it, you can switch to something else
08:53But personally for me, it is my favorite for larger Python projects and it is literally designed for Python, hence the name PyCharm
09:00Anyways, let's get started here
09:00So, what I've done is I've opened up a new folder and I've just called this AI search agent
09:05You can call this anything you want
09:07And then from this folder, we're going to initialize a new UV project and create our virtual environment in Python
09:12So in order to do that, I'm going to type UV init and then dot in my terminal
09:17This is going to initialize a new UV project
09:18And from here, we're going to install the dependencies that we need
09:22So I'm going to type uv and I'm going to add lang chain
09:29Okay
09:29and then lang graph and then we're going to install lang chain and then dash open ai because we need to use gpt here and then we're going to install python env
09:39Okay, these are the four dependencies that we're going to need to install
09:44So let's go ahead and press enter here and they all get installed in our environment
09:47Now feel free to use pip or any other virtual environment that you want
09:50But if you want to use UV and you're not familiar with it, I'll leave a video on screen that teaches you how to use UV because it's kind of the standard now
09:57It's very fast for managing dependencies and environments in Python
10:01Okay, so now that we've got UV installed, we're going to start setting up our project
10:05So, there is quite a bit of setup and there's going to be a lot of code here
10:08If at any point you're getting lost or you just want to copy something that I'm writing, you can do that by clicking the link in the description
10:15There'll be a GitHub repository that contains all of the code for this project
10:18So, what I'm going to do now is I'm going to make a new file inside of this folder called env
10:22Now, this is going to store some environment variables that we need
10:26I'm just going to ignore this for right now
10:29Specifically, our OpenAI API key and our bright data API key, which we're going to get later on or in a few minutes
10:33So, we'll just start by writing the variables that we're going to need
10:37So, first is going to be bright data API_key
10:42Okay, we'll fill that in later
10:42And then next is going to be the open AI API_key
10:48And again, let's spell open correctly
10:50We can get that later
10:50Okay, so we have our environment variables defined
10:53Now, we're going to go into our main.py
10:55py file
10:57Just create a new one if you don't have one here and we'll start writing some code
11:00Now, PyCharm is also prompting me just to configure my interpreter
11:03So, let me just select the correct one and then I'll be right back
11:06All right, so I've configured the correct interpreter and now what I'm going to do is start scaffolding my project
11:11Now, when I say scaffolding, what I mean is I'm going to write all of the functions and logic that we'll eventually implement
11:16But for now, I'm going to just kind of connect it together so that we understand the architecture and the flow that we're going to follow along with
11:23And then once we have that, it's a little bit easier to go write each individual function
11:28This is typically how I plan larger projects
11:30So I'm going to kind of walk you through my thought process in this video and you'll see how we plan it out
11:35So first we're going to say from So let's do this correctly
11:39Env import load.env
11:42This is going to allow us to load in the environment variables that we've defined in this
11:46Envile
11:46I'm going to say from typing and I'm going to import annotated
11:51Okay
11:51I'm gonna say from lang graph and then this is going to be dotg graph
11:59I'm going to import the state graph and the start and the end nodes
12:05Now again if you're not super familiar with lang graph essentially this allows us to build a graph which is a bunch of different nodes that are connected to each other and to flow some state or some data through that graph where each kind of node in the graph can modify or update that data
12:19So we're going to have some state which is going to store all of the information that our agent needs to have access to
12:25And as we run through these different stages or nodes in our graph, we'll be populating that state where then at the end of our graph, we have this final answer which we can present to the user
12:35So it's kind of a really unique way to build AI agents in a bit more of a kind of predictable flow
12:41Rather than just giving a set of tools to a model and letting it go crazy, we actually kind of walk through this manual process that's a lot more consistent where we update this state kind of stage by stage
12:52Okay, so next we're going to say from lang graph and this is going to beg graph
12:59Okay, dot message we're going to import add messages
13:02We're then going to say from langchain dot and this is going to be chat_models
13:09We're going to import the init chat model which is a really quick way to initialize an LLM in langraph
13:13We're then going to say from typing extensions and we're going to import the typed dictionary
13:21Okay, these are just some typings that we need in Python
13:23And then we're going to say from pi dantic
13:25Okay, and we're going to import the base model and the field
13:34And then we're going to say from typing import list
13:37And actually I realized we can just put this up here because we need list and annotated and they come from the same package
13:46Okay
13:46So that's most of our imports
13:49Later we'll import a few other things but for now we can start with this
13:52And then we're going to call this load.env function
13:53When you call.load env file and it loads these variables for us that we can start using them inside of our Python code
14:02Okay
14:02Now, next step, we're going to say llm is equal to init chat model
14:07And for the chat model, we're going to put the name of the LLM we want to init, which in my case is just going to be GPT40
14:12Now, you can put pretty much anything that you want here
14:16You just need to make sure that if you put a different model from another provider that you pass the correct API key in this file
14:21So, automatically when we try to load GPT40, Langraph is going to look for the presence of this OpenAI API key variable and then use that as our API key
14:31Okay, so we just put GPT40
14:34You can put a newer model, whatever you want
14:35Just make sure you have the correct API key
14:37Okay, so now that we've done that, what I want to start with is actually writing out the state that we're going to pass through our graph
14:46When we have the state, we'll kind of understand the data that we need to come up with and find
14:50And then I can start creating this graph and making the connections between these nodes
14:54And then we can start actually kind of populating the graph by writing the different implementations
15:01So for now we're going to say class state and this is going to inherit from the typed dictionary
15:05Okay
15:05Inside of here we're going to start by having a list of messages
15:10Now these messages are essentially the messages that our user is sending into this graph that we'll then process and start getting the information for coming up with an answer for
15:21So we're going to say messages is annotated and then this is going to be list and add underscore messages
15:25When we do this, we mean that okay, messages is type list
15:30And when we want to add a new message, we call this add messages function, which will uh essentially modify the messages for us
15:36Okay
15:36Next, we're going to have the user question
15:41And this is going to be string or none
15:44Okay
15:44Then we're going to have the Google results
15:49And this is going to be string or none
15:52And you'll notice that all of these are going to be or none because at some point in time, we may not have these results
15:57and then later we'll populate it
15:59So then next we're going to have the Bing results
16:01This is going to be string or none
16:03Then we're going to have the Reddit results
16:06Again, string or none
16:11Okay
16:11Then we're going to have the selected_reddit URLs
16:18This is going to be list of type string or none
16:21Now, the reason why I have selected Reddit URLs is because we're going to get a bunch of results from Reddit
16:28And then we're going to pass these results to an LLM where it's going to select which of these URLs we actually want to process further just to avoid us looking at data that we don't need
16:37Then we're going to have Reddit_post data which is going to be the data for those selected URLs
16:42This is going to be list or none
16:44We are then going to have the Google_analysis
16:51This is going to be string or none
16:51This is the kind of LLM analysis
16:53After we get our results, we're going to have the Bing analysis which is string or none
17:00And then the Reddit analysis which is string or none
17:02And then finally the final answer which again is string or none
17:07Okay
17:07So this is essentially our state
17:10This is what we're going to be flowing through the graph
17:12And we'll start populating these one by one as we kind of go through all of the nodes that we have in our lang graph
17:18All right
17:20All right
17:20Now, that's great
17:20What we need to do next is we need to start defining the nodes that we have in our graph
17:25So, if we look here, this is what we need, right? We need to do Google search, Bing search, Reddit search, analyze the Reddit post
17:30All of these are essentially just functions that need to execute in some order
17:33So, what we're going to do now is we're going to write all of these empty functions without the implementation
17:39And then later, we're going to implement all of these functions
17:42This way we can build this flow and then later we can go and we can actually add all of the different pieces and kind of test it step by step
17:48So what I'm going to do is I'm going to start defining a bunch of functions
17:52So the first function that I'm going to have is going to be called Google search
17:56So I'm going to say define Google search and then for all of these functions what they take as a parameter is simply the state
18:03Okay, so we say state and then that's going to be equal to this right here
18:07they take in this state and then what they're going to do is return something that modifies the state
18:12So for right now, we're just going to say return
18:14We don't need to return anything right now
18:15And we're just going to keep making a bunch of functions with all the different operations that we need to perform
18:19So after we have Google search, then we have the Bing search
18:24Okay, that's the next function or the next note
18:26Then we're going to have the Reddit search
18:30And again, we're just going to populate all of these and then finish them later
18:33Okay, then we're going to have analyze Reddit posts
18:35So let's go analyze Reddit posts
18:39Okay
18:39Next function is going to be retrieve Reddit posts
18:43So let's change this
18:45Retrieve Reddit posts
18:51Okay
18:51And I'm just going to add some spaces between here because you can see PyCharm is kind of linting this and telling us that we should have two spaces between our functions
18:58So that's fine
18:59After this, we're going to have analyze Google results
19:01So analyze Google results
19:06We need to analyze the Bing results as well
19:09So in fact, let's just copy this function and paste this down here
19:13Analyze Bing results
19:17Okay
19:17Then we need one more for analyzing the Reddit results
19:23Okay
19:23So Reddit results are a little bit different than the Reddit posts
19:25So we're going to have Reddit results and then we're going to synthesize this
19:29So, we're going to say synthesize_analysis like that
19:35And then this is going to actually take all of the results that we had here and synthesize it into one larger result
19:41Okay
19:41So, we have a bunch of functions
19:44We're almost done with functions
19:46What we're going to do now is we are actually going to create the graph where we connect these nodes together
19:52So, again in Lang graph essentially we have nodes
19:53Nodes are really just operations or functions
19:55So we've defined now what all of our operations are going to be
19:59What we need to do though is we need to connect them to one to one each other
20:03Sorry
20:03So we know that we start for example with Google search then we do the Bing search then we go to Reddit search then we analyze this and we need to just create these connections or create the graph
20:14So what we're going to do is we're going to say our graph builder is equal to and this is going to be state graph
20:18And for the state graph we simply pass our state
20:23Okay
20:23And now that we have that, what we're going to do is we're going to add all of these nodes into our graph
20:29Now, the way that you do that is the following
20:30You say graph_builder add node
20:33And then what we're going to do is give the node a name
20:35So we're going to say something like Google search
20:38And then this is going to be pointing to the Google search function
20:42But make sure you don't call the function
20:43All right
20:43Now, this is a little bit tedious, but essentially you just need to give a name to every function or every node that we have
20:48So you see like the next autocomplete is we're going to add the Bing search and that's going to be the Bing search function
20:54We have the Reddit search that's going to be the Reddit search function
20:57And I'm just going to keep going here and using my autocomplete
20:58So we're going to have analyze Reddit post
21:01That's going to be to analyze Reddit post function
21:03Analyze Google results
21:05And you get the idea
21:05And I'm just going to go through here and analyze results results is definitely not the name of a function, is it? Um okay
21:11If that is then I definitely spelled something wrong
21:14So we don't want analyze results result
21:17We want analyze Reddit results
21:21Okay, so let's fix that
21:21And let's go here to analyze Reddit results and analyze Reddit results
21:29Okay, good job
21:29We caught that
21:31And then we have the last one, which is the synthesis
21:34So, let's add that here
21:36Okay, it looks like we're actually just missing one here, which is retrieving the Reddit post
21:40So, let me add this
21:42We're going to say graph builder.add add node and this is going to be be retrieve okay Reddit post like that and then we're going to fix this to be retrieved_reddit post
21:58So we analyze the Reddit post we retrieve the Reddit post and then we have the rest of them here which I think is all good
22:03Okay
22:03And ignore the yellow highlight
22:05We're going to fix that later on
22:06It's just because we're returning none from these functions
22:08All right
22:09All right
22:09So at this point, what we've done now is we've created the nodes
22:11So they exist, but they're not yet connected
22:14So now that we've created them, we need to connect them to one to one each other
22:18Sorry
22:18So the way that we do this is we can say graph_builder
22:23And we can say add edge like this
22:23And an edge is a connection between the nodes
22:28And we're going to connect the start to the first node, which is going to be our Google search
22:33So what this says is okay when we start the graph the first thing that we're going to do is we're going to go to Google search
22:39Now what we're going to do is we're going to connect the start to multiple of these
22:44So we're going to connect to Google search as well as Bing search as well as the Reddit search
22:48So this way what will happen is at the exact same time we'll execute all three of these operations
22:55So we'll run them in parallel
22:55So as soon as we start or as soon as the user gives us some message or some request, we searched on Google, we searched on Bing and we searched on Reddit at the same time
23:05Okay, cool
23:05So next thing we're going to do here is we're going to connect the next steps
23:09So after Google search, what do we do? After Bing search, what do we do? You get the idea
23:15Okay, so in order to do this, we're going to say the graph builder added
23:20And the first edge that we're going to add might seem a little bit weird, but from Google search, we're actually going to connect the analyze Reddit posts
23:28Okay, let me make this a little bit smaller so you guys can see this
23:30Now, the reason we're doing this is because if we follow our architecture, right, after we do all of the searches, we need to wait a second to get all of the Reddit results before we can move any further
23:41So, all of these connect to the analyze Reddit post, which is what we're doing right now
23:46Then, we'll retrieve the Reddit post
23:47then we'll go and do all of this analysis
23:50Now, there's actually some ways that we can make this a little bit more efficient, but for right now, this is just a simpler architecture that I want to follow
23:56Um, you probably know what I mean if you're thinking like, okay, how can I make this a bit more efficient? But for right now, I don't want to make it too complex
24:03So, we're just going to go with this
24:05Okay, so after the Google search, we go here
24:08After the Bing search, we're going to go here as well
24:11So, we kind of wait at this stage
24:13And then, of course, after the Reddit search, we need to go here, too
24:17Okay, so we've now made this next connection
24:19So we have start to Google search, start to Bing search, start to Reddit search, and then from each of these we wait at the analyze Reddit posts
24:26Okay, now after we analyze the Reddit post, what we need to do, so let's put one more in here, is we need to retrieve the Reddit post
24:34So we're going to go analyze Reddit post to retrieve Reddit post and add that edge between those two
24:40All right
24:40Now, the next thing that we're going to do is we're going to say graphbuilder.add edge
24:46Let's get rid of all of this
24:46And we're going to start now from the retrieve Reddit post
24:50And after the retrieve Reddit post, what we're going to do is we're going to start analyzing all of our results
24:56So we're going to say analyze
24:58And then the first thing that we're going to do is the Google results
25:01So underscore Google results like that
25:05Let's spell analyze correctly
25:07Okay
25:07Now let's copy this and go down here
25:09Now, after retrieve Reddit post, we are also going to go and analyze the Bing results
25:13Okay? And then we're going to go one more down here
25:20After we retrieve the Reddit post, we're also going to analyze the Reddit results
25:25All right? So, hopefully this is making sense
25:27But again, if we go back to the architecture, we wait for the Reddit post, we retrieve them
25:31Then after we retrieve them, we go and we analyze the Google results, the Bing results, and the Reddit results
25:36That's what I've just written, right? We go Google, Bing, Red
25:40So, analyze those three at once
25:42Okay
25:42Now, after those three, what we need to do is we need to synthesize our analysis
25:48So, we're going to do this again
25:51We're going to add another edge
25:52Now, we're going to start from analyze Google results
25:54And the end key here is going to be the synthesize analysis
26:00Then, we're going to copy this
26:00And I'm just going to copy it twice because we're going to need it here
26:03After we analyze the Bing results, we're also going to go there
26:08And after we analyze the Reddit results, we're going to go here as well
26:14Okay, so those three then connect to this next node where we're synthesizing everything
26:18And then lastly, we add one more
26:19So we say graph builder added edge and we're going to say from the synthesize analysis, we're going to go to the end
26:27Okay, so this is how you set it up
26:31You need to always have a start key, which we do right here, and an end key or an end node
26:35And then this creates this graph that you just saw in this diagram
26:40Okay
26:40Now, after we create the graph, what we can do is compile it
26:44So, we're going to say graph is equal to graph_builder and notbuild, but compile
26:50This is going to actually execute the graph for us
26:53So, we're able to run it
26:54And then what we can do is we can essentially pass a message to this graph
26:58it will run through all of the different nodes
27:00The state will get updated and then we can print out that state
27:03So what I'm going to do is I'm going to write the function that would allow us to execute this graph
27:08And then of course before we can do that, we need to start writing all of these different functions
27:12So we're going to make a function here
27:13I'm going to call this run chatbot
27:14Okay, what this is going to do is start executing our uh what you call agent so that we can actually run through this graph
27:22So, what we're going to do is we're going to do a print statement and we're just going to say a multi- source research agent like that
27:31I'm going to print a line and I'm going to say, you know, type exit to quit
27:37Okay
27:37And then back slashn
27:42All right, that's good
27:42And then what I'm going to do is have a while loop
27:45So, I'm going to say while true
27:45And we're going to keep asking the user to give us some input
27:48So we're going to say user input is equal to and then we're going to say input I'm going to say ask me anything colon like that we're going to say if the user input
28:01equal to exit then print by and we can break the loop
28:05Okay
28:05Otherwise what we're going to do is we're going to say state is equal to and we need to initialize a kind of starting state
28:11So in order to do that, we could say state is equal to the following and we can say messages and then for the message we just need to put the message that we want the uh bot to reply to
28:22So we're going to say roll is user and we're going to say content is the user input
28:32Okay, we're then going to say that the user question is the user input
28:34We're going to say the Google So we need to actually have this in a string
28:40The Google results is just equal to none
28:44And then we're going to do the same for all the rest
28:45So the Bing results is equal to none
28:46The Reddit results is none
28:48The Google analysis is none
28:48The Bing analysis is none
28:51The Reddit analysis is none
28:53Final answer is none
28:55And then of course there's a few other ones that we missed
28:56So let's go here
28:58After Google results, Bing results, Reddit results
28:59We also have the selected underscore Reddit URLs
29:04Okay, this is going to be none
29:07We then have the Reddit post data
29:10So let's do this
29:13Reddit_post data and that is none as well
29:15Okay
29:15And I think that's all of the state that we need
29:19So again, kind of at the beginning, we need to initialize the state that we're going to be passing through the graph
29:23So that's what we've just done
29:24We've plugged in the user input
29:26So the question that they've asked us and then what we're able to do is start running the graph
29:29So what we can do here is we can do a print statement and we can say back slashn and then something like starting parallel okay and let's go here research process dot dot dot then we can do another print and we can say something like launching Google this is just for logging by the way but I think it looks nice bing and reddit searches dot dot dot okay I'll do a back slashn here as well to kind of separate this out
30:03And then I'm just going to print a dash* 80 uh just so that we get kind of some separation here between what is appearing
30:12Okay, so actually I just missed something
30:14So up here what I'm going to do is I'm going to say the final state is equal to graph.invoke
30:22I'm going to invoke the graph with my state
30:24And then what I'm going to do down here is I'm going to say if the final state and then I'm going to say get and this is going to be the final answer
30:33So if that does exist then I'm going to print out the answer
30:35So I'm going to say print
30:37Okay
30:37I'm going to do an fstring
30:39I'm going to say back slashn
30:40I'm going to say final answer like that
30:44I'm going to say back slashn again
30:46And then I'm going to put the final state
30:48Okay
30:48Get final answer like that
30:53and then we'll put a back slashn
30:56Okay, so essentially what I'm saying is all right, we're going to invoke the graph
30:59This is how you invoke it
30:59We pass our initial state
31:01Again, don't worry too much about that
31:02We'll fix that later on
31:03We say if the final state does contain a final answer, then we'll print out the final answer and it will just print kind of a separation here so that if we run this again, we know, you know, which run was which
31:13Then lastly, we need to execute this function
31:15So we can say if name is equal to main, then we can run the chatbot
31:19And then that actually would be a finished program assuming that all of the nodes were completed which of course they're not and we're going to need to write
31:27So let me quickly just kind of summarize what we've done here
31:31I just want to zoom out a little bit so you guys can read this
31:34Essentially we started with all of our imports
31:35We loaded the environment variable file
31:37We initialized our LLM which we still need the API key for which we're going to get in one second
31:42We created our initial state
31:42Okay
31:42And then what we did is we kind of stubbed all of these different operations or nodes that we're going to have in our graph
31:50So Google search, Bing search, Reddit search, analyze Reddit post, retrieve the Reddit post, analyze Google results, analyze Bing results, analyze Reddit results, synthesize the analysis, and then we created the graph
31:59So we added all of the nodes where we connected functions to the node name
32:04That's essentially what we did here
32:04We then added all of the edges
32:06So we started with these three running in parallel
32:09We then kind of connected them to this analyze Reddit post node
32:12We get the Reddit post here after we analyze them
32:16And then we go through the rest of the flow until we eventually synthesize all of the results
32:20We then have this uh while loop that just allows user to type something in and essentially run it through our graph
32:25And that is the lang graph component kind of done
32:27What we need to do now is we need to start updating the state as we go through these various nodes
32:33So let's get into that
32:35And that's going to allow us of course to start searching the web using the SER API, all of that kind of stuff that I'm going to show you
32:40So for now, because I want to be able to test this step by step, I'm going to start filling in some of the outputs that we're going to have from these functions so that even if they're not fully complete, we'll be able to execute the graph and kind of test the Google search first, then the Bing search and see what results we're getting
32:56So from our Google search function, I'm going to say the user question is equal to state.get and this is going to be the user question or an empty string
33:06Okay, so because I have state in all of these functions, I can pull out the state
33:13Then what I'm going to do is I'm going to have a print and I'm going to say f string
33:16So, and this is going to be searching Google for and then we'll do a colon
33:22So, searching Google for the user question
33:25Okay, I'm then going to say the Google results are equal to an empty list and later we'll actually get the uh Google results, but for now we'll just make it an empty list
33:35And then I'm going to return the following which is Google results is equal to the Google results
33:44Now whenever you return something from these functions it needs to match what you have in the state
33:48So in this case we have Google results matching with our Google results
33:52So this Google results will get updated to be equal to whatever this is
33:57And then in the next function we have access to these updated Google results
34:01That's how the state kind of flows through here
34:03you return a partial update to the state from one of these nodes and then it gets updated here where it's continually passed to all of the next nodes in the sequence
34:13Hopefully that makes sense
34:13But that's kind of the idea
34:15Now for the Reddit search or the sorry the Bing search, it's effectively the same thing
34:19So I'm just going to paste this here and rather than searching Google, I'm going to search Reddit or not Reddit, Bing
34:23I keep messing these up
34:26And then rather than the Google results, this is just going to be the Bing results
34:30So change this change this and change this here
34:38Okay
34:38And then for the Reddit search, again, it's pretty much going to be the same thing except just named Reddit
34:43So, we're going to paste this here
34:45We're going to say searching Reddit
34:48And then just update all these variables to say Reddit
34:52Okay
34:52Reddit and Reddit like that
34:56Okay
34:56Now, we're going to go to the analyze uh Reddit posts
35:01From here, essentially what we're going to do right now is just return some fake data
35:05So we're going to say the selected_reddit urls
35:08And for now, this is just going to be equal to an empty list
35:11Later, we can populate that, but for now, that's all we need
35:15Now, same thing when we talk about the Reddit post data, we're just going to say return Reddit post data is equal to an empty list
35:22And then for the analysis, same thing
35:24We're going to return
35:26Okay
35:26And this is going to be the Google analysis
35:29And for now, just going to return an empty string
35:36And then we can do the same thing for Bing
35:38So return the Bing analysis empty string
35:42Okay
35:42Return the Reddit analysis empty string and then the final answer
35:49So we're going to say return final answer
35:55Okay
35:55And then empty string
35:55just so that all of these functions work properly and they return the correct format
36:00Okay, cool
36:00So, I just saved this now and what we can actually do just to test and make sure that the logic is set up correctly is we can just run this file
36:07It should ask us to type something in and it should just give us no response
36:12So, what I'm going to do is just press on run here and we'll see if we get any errors
36:16It should just prompt us to type something
36:18So, it says ask me anything
36:20So just go hello and then you can see that it just kind of gives us this output and then doesn't say anything and says ask me anything
36:25So that to me means that this is working again type hello and you can see that it just kind of doesn't give us anything but we get some output here and that's actually exactly what we were looking for
36:35So I'm going to stop this here because it means the flow is working properly
36:38And I did notice that we have a few spacing issues
36:40So let me kind of fix that here
36:43Looks like yeah we randomly printed out a quote
36:45So, let's get rid of that and fix the quote here
36:50And I think we're kind of good to go
36:53So, the next step is going to be to set up our LLM as well as to set up Bright Data to start doing these search operations
36:59I want to start by searching Google, then searching Bing, and kind of walk through these and do one search operation at a time so you understand how they work
37:06So, what we're going to do is we're going to go and get these API tokens
37:08So, we're going to create a Bright Data account in an OpenAI account
37:12Let me go to my browser and let's set that up
37:14All right
37:14All right
37:15So, let's start by getting our OpenAI API key so we can use GPT and then we'll get the bright data one
37:20So, what we're going to do is go over to platform.openai.com
37:25From here, we can just go to our settings, then to our API keys, and we can create a new key
37:30So, from here, I'm just going to go with AI agent as the name
37:35Okay
37:35And then obviously, you don't want to leak this key
37:36So, I'm going to copy it and I'm going to paste it here in my environment variable file
37:43Okay
37:43And now that we have that, we need the Bright Data credentials
37:44So, if you don't already have an account, you're going to need to create a new one on Bright Data
37:49I'll leave a link below in the description and you should be able to get some free credits
37:52So, you do not need to pay to use this for the tutorial
37:56Now, I quickly want to show you a few of the services that we're going to use here from Bright Data because they have a lot of options when it comes to getting web data specifically for AI agents
38:04So, for example, they have a chat GPT scraper, right? where you can actually scrape the conversations from chat GBT, the responses, the user queries, etc
38:11We're not going to use that here because we don't really need it for this specific tutorial, but in other ones, it's quite useful
38:19Now, we also have, let's go here, social media scraper
38:22So, this is the one that we're specifically going to use to scrape Reddit data
38:25You can also get stuff from Facebook, Instagram, Tik Tok, YouTube, which is notoriously very difficult to scrape
38:31If you've ever tried to build your own scrapers before, you've likely seen that it's very complicated to actually get this data and you get blocked by captas, IP bands, uh, etc
38:41Whereas Bright Data can actually overcome and bypass all of that for you and just give you the data in a very easy format
38:47So, for example, you can get Instagram profiles, posts, X, LinkedIn
38:51In our case, we're using Reddit, which I think makes a lot of sense for this particular agent, but obviously you can pick pretty much anything you want
38:57And then we have the SER API or the search engine API where we can really quickly scrape all of the major search engines like Duck.go, Google, Bing, etc
39:06This also works for things like Google flights, right? Uh, and all of those other services that come from those search engines
39:12Yeah, like maps, images, hotels
39:13Pretty cool
39:16I've done some other projects in the past where I've used this
39:17And again, for this one, we're just going to use the standard kind of Google Bing search engines
39:22They also have things like a web archive
39:23So, for example, if I go to the documentation here, you can see that you can actually scrape all of the previous web data
39:29So, you can get like years back and you can kind of see trends and historical data
39:33Again, not going to use that for this video, but we could add that if we wanted to make it more complex
39:38Okay, so for now, we need to make a new account or log into our existing account
39:42So, go to the link that I have in the description
39:44I'm going to log in because I already have an account here
39:47For you, you are likely going to create a new one
39:49So, once you've signed into your account, you should be brought to a page that looks like this
39:53They actually recently added this feature where you can just ask the AI here and it can tell you how to do what you want to do
39:58Uh I'm not going to use that though
40:00What I'm going to do is go to proxies and scraping from the left hand side here
40:04And what we're going to do is create a new SER API
40:05Now of course there's a lot of other features here as well like web scrapers
40:09We'll use this later to actually collect the Reddit comments and the posts as you can see that I was kind of doing already
40:15But for now, we go back to proxies and scraping and we're going to go to add up here and we're going to create a new SER API, search engine API
40:22Okay
40:22So, press on this
40:24From here, we can give it a name
40:26I'm just going to call this AI agent uh two because I already have one called AI agent
40:30Can give it a description if you want
40:32And then in this case, I'm just going to leave this at standard, but you could go maximized if you care about actually retrieving the ads
40:38Okay
40:38There's a few advanced settings as well
40:40We don't really need to modify any of those currently
40:42And we can just go ahead and press on add again
40:47You should have some free credit
40:48So this should be free to use for you
40:50And then later obviously you can pay for it if you want to use the service
40:51Okay
40:53So I'm going to go yes create this new zone
40:55Once this is created it should give us access to an API key which we can actually see here and show us how we can call this API
41:02So notice here we have method API and then we can do for example Python and it gives us an example of how to call this
41:08What I'm going to do for now is I am just going to copy the API key which is right here
41:15And we're going to take that and put that into our uh file into our environment variable file
41:19So let's go here and paste the API key
41:21And obviously don't leak that to anyone
41:23I'm going to delete that after this video
41:26And then we'll be able to start using this service, the search engine API
41:28Now if you go to the playground, you can actually mess around with it here and you can uh kind of test out your different searches
41:36So for example, we could search all of these different engines here
41:39We can choose the keyword that we want to search
41:41And then there's a bunch of other information that we can add
41:44So we can search a specific Google
41:46So dot, you know, France.AE dot whatever, right? And then we can add all these other settings like do we want to look on desktop or mobile? Do we want to add specific headers? Do we want to actually get a page nated response? Do we want geoloccation? There's all these different parameters that we can add
42:01And there's some examples that you can view here as well on exactly how to do this
42:05In our case, it's going to be pretty simple
42:07We're just going to search Google
42:08Um, and that's kind of it
42:10So, what we'll do from now is we'll go back to main.py
42:12And I'm actually going to make a new file where I'm going to call this webcore operations
42:17And inside of this file, which will be a Python file, I'm going to start implementing all of the operations related to the web scraping and to using the bright data service
42:28So, inside of here, we're going to start with just our basic search
42:31So, we're going to be searching Google
42:33But in order to do that, I'm going to set up some reusable functions that will make our life a little bit easier in the future because we'll be sending quite a few requests over here to the bright data SER API
42:41So what I'm going to do is I'm going to say from enenv import load.env again because we need to import our environment variables
42:50I'm going to say import OS import request and I'm going to say from URL lib.parse parse import the quote underscore plus which is going to allow us to turn a normal string into a string that we could include in a query parameter in our URL which you will see in a minute
43:12For now, I'm going to call load.enb to load theenv function
43:14And I'm going to make a simple function here called make API request which will just be a reusable function that we can use anytime we want to send a request to bright data so we automatically can include the correct headers for our authentication
43:28So I'm going to say define_make API request like that
43:34I'm going to take in the URL and starst star quarks like that
43:38Here I'm going to say API_key is equal to os.get env
43:41And we're going to get the bright data API key like that
43:48Okay
43:48So we're going to get the API key and we're going to create a set of headers because we need to send these headers to tell bright data who we are
43:54So we're going to say authorization is equal to fstring and then bearer space and then our API key and then we're going to say the content dash type is going to be the application JSON because that's what we want to get back
44:07Okay
44:10From here we're going to do a simple try accept block where we send a request to whatever the URL is that was provided here
44:16So we're going to say try and this is going to be response is equal to request.post
44:23We're going to post to the URL with headers equal to headers like that and pass our star star quarks
44:29We're then going to say response.rafor status
44:34What this means is we're going to raise an exception if we don't get an okay status
44:37And then otherwise we're going to return the response.json
44:42Okay
44:42Then we're going to say accept and this is going to be request.exceptions exceptions dot and this is going to be the request exception as E and we're going to say print like this an F string and we're going to say API request failed and then we'll put E inside of parenthesis or inside of braces and then we can return none because this didn't give us a response
45:06Then we can have another exception
45:06So we can say except any general exception as E
45:10We can say print f unknown error and then we can print out e and again return none
45:18I'm just doing some more advanced exception handling here just so that if it's related to the network request we can handle that
45:24If it's not related to the network request then we deal with it here
45:27So we kind of know which exception or what error was actually causing the problem
45:30Okay
45:30So now we have a general function that can send a request to brightite data
45:33What we need to do next is implement our SER function
45:38Okay
45:38So we're going to say define and this is going to be SER search
45:43And what this is going to do is take in a query and an engine which by default right now is going to be equal to Google
45:49Now I'm going to write this kind of dynamically because this will allow us to actually search any search engine that bright data supports
45:55So something like Bing, Google
45:57So we can reuse this function multiple times
45:59All right
46:02What I'm going to do is I'm going to say if the engine is equal to Google, then I'm going to say the base URL is equal to https/google.com/arch
46:17And this needs to be ww.google.com/arch
46:21We just add a new line here
46:21We're going to say l if the engine is equal to bin or not bin, bing
46:26Then the base URL is https www.bing.com/ bing.com/arch and then else we're going to raise an error and we're going to tell them hey this engine is not supported
46:36So we're going to say raise value error and then unknown engine and then whatever engine they passed
46:41Okay
46:41Now we're going to say the URL is equal to https slash and this is going to be api.brightdata.com/ request
46:52Okay, because this is where we're going to send the request and we're going to pass essentially the search URL that we want to search and get the data back from
46:59Now, we're going to say our payload is equal to zone and the zone is going to be the name of the zone that we created, which is AI agent 2
47:09So, if we go back here and we look at our overview, we should be able to see the zone name
47:13You can see it's right here, AI agent 2
47:15Okay, so that's our uh zone name and you can kind of see the information down here as well
47:19So, anyways, we need to pass the zone
47:21We also need to pass the URL
47:23So the URL is going to be the following
47:26We're going to put an F string
47:27We're going to put our base URL, which is the search engine that we want to search essentially
47:33And then we're going to say question mark Q is equal to and we're going to say quote unl
47:43Now what this is going to do is it's going to take whatever the user typed in
47:46That's going to be our search string
47:47It's going to turn it into a format that we can actually pass correctly in a query parameter for this URL
47:52And then we're going to put an and we're going to say BRD_JSON equals 1
47:58Now what this means is bright data JSON enabled
48:00So essentially we want to get our responses back in JSON format
48:05So Bright Data is actually able to parse all of the responses from the search engines and then return it to us in a digestible format
48:10In this case, JSON
48:13And then we're going to say the format is raw like that
48:15Okay
48:15So this is the payload which is essentially how we send this search request
48:20This will hit the search engine API and then it will give us back a response
48:25So what we're going to do here is we're going to say the full underscore response is equal to underscore make API request
48:34We're going to pass our URL right which is the URL right here
48:36So that's where we're sending the request to
48:38And then we're going to say actually the JSON is equal to payload which will be another query parameter that we pass there along with our request
48:47We're going to say if not full response then we're going to return none
48:54Otherwise what we're going to do is extract data out of this response
48:57So the bright data response is going to give us a ton of information
49:01It's going to give us the sponsored post, the organic post
49:05It's gonna give us a bunch of stuff, but we only care about a few sections of that response
49:08Now, if you want to look at this response, you can just mess with it right here
49:12You can go to the playground, right? And we can kind of run this request and see the result that we get
49:17But what I want to do is I just want to pull out a few pieces of information
49:22So, right, it's giving us the full kind of preview of the page
49:24We can actually look at the JSON format and you'll see it has like general, input, navigation, it has all of these other fields, right? It's a very long response that we get, but I only care about part of the response
49:37Now, the part of the response that I care about are to the organic results and the knowledge that Google pulls here
49:43The knowledge is like a quick summary of the uh information that you search for that you've probably seen before if you've done, you know, a Google search
49:49So, what I'm going to say is extracted data is equal to this and I'm going to say knowledge is equal to the full response.get get and I'm going to get the knowledge field
50:05Now, if that doesn't exist, I'm just going to get an empty set of braces or an empty dictionary
50:08Then I'm going to get organic and this is going to be the full response
50:12And then same thing, organic except here, this is actually going to be a list
50:15Now, the reason I know this is because before the video, I was obviously preparing
50:20I looked through the response structure and these are the two fields that I care about
50:22If you want to see the entire response structure, then feel free just to print it out and you can see all the information that it gives you
50:28Here, we're just narrowing down the data
50:30that we only get the important stuff
50:32But if you wanted all of the data or something else for a different use case, then of course you can get that from this API request as well
50:38And what's interesting here is that you can run this as many times as you want
50:41It's very scalable
50:43So you can run this, you know, hundreds of times, thousands of times with different requests
50:46And it returns very, very quickly because it's already indexed by bright data
50:49So I'm going to return the extracted data here from this function
50:55And then that should be it for this first function where we're essentially just calling this kind of search function
50:59All right, so that's pretty much it, at least for right now
51:04What I'm going to do is I'm going to go back to main
51:05I'm going to import what we just wrote and then I'm going to call that from one of our functions so we can test it out and make sure it works
51:10So I'm going to say from web operations import and we're just going to import the SER search
51:16So now where we go to Google search the Google results are actually going to be SER search and then the user question and we're just going to say the engine is equal to Google and then that should give us back the Google results
51:31Okay
51:31So for now what we can do is we can just print the Google results so we can see if we're actually getting anything at all
51:37And while we're at it we might as well just do the same thing for Bing because it's going to be the same thing just with a different engine
51:44So let's go SER search
51:44We'll go user question engine is equal to Bing and then same thing we can print the Bing results and we can do an initial test here to see if this is working
51:54Okay, so let's run the file and let's go invest in video
52:00Okay, and it says searching Bing, searching Google
52:02Wait a second and then we should get the results
52:04And you can see no knowledge popped up for this one
52:07That's okay
52:07And then for organic there's a bunch of links, right? So it gives us kind of all of this data popping up related to those results and you get descriptions
52:16So I start thinking of Nvidia stock etc etc
52:18Now of course there's a lot more stuff that we can extract from here but for now that is good and it gives us kind of the top results on Google and we can read through the descriptions the links etc
52:29Okay, so let's exit out of that and let's continue because now we have the SER API functioning
52:34And I'm just going to remove the print statements because now that we know we're getting the correct results, we don't really need anything more
52:41All right, so we have Google search, we have Bing search
52:43Now what we want to do is we want to implement the Reddit search
52:47So for the Reddit search, it's a little bit different and that's going to require us going to web scrapers
52:52So from here, we're going to go to new
52:54We're going to go to browse scraper marketplace and then we're going to search for Reddit
53:00Okay, so it's going to take a second
53:02We're going to press on the result for scrapers and then this is going to give us a few options that we can use to actually scrape Reddit and get the real lifetime data
53:09Now you can build your own scraper if you want, but a lot of them that you need are already built for you and you can just call them like an API which makes it very easy to actually download and get the data quite quickly
53:20And in our case, there's two main things that we want to use here from Reddit
53:22We want to discover posts by their keywords because we want to search essentially on Reddit
53:28And then after we search, we want to get all of the relevant post URLs and download all of the comments
53:35So you can see we have Reddit's comments collect by URL
53:37Then we have Reddit post discover by keyword
53:39So we're going to use both of them
53:40First, we collect the post
53:42Then we get the comments from the post that we care about
53:44So let's go to Reddit post here
53:46It's going to say scraper API
53:47So I'm going to go ahead and press on next
53:49And it's going to create this scraper for us
53:51Now here we have collect by URL, collect by keyword, collect by subreddit URL, right? Like and we can get the comments as well
53:57And we can kind of run through this and see how we use this scraper, all of the fields that we can pass to it and all of the fields that we'll get back
54:06So it shows us what the response structure looks like
54:09And then if you go to the API request builder, it shows us how to build this API
54:12Now our API key will be the same
54:15The thing that's going to change is the URL that we need to hit here
54:19And this is where I want to go in and talk about our management API
54:21Okay, so when we use this um scraper, essentially what's going to happen is we're going to send a request and then bright data is going to go using its scalable network and start collecting all of that data for us
54:35Now, it's not going to be available instantly because it needs to actually access it in real time from Reddit
54:42So what's going to happen is we're going to create something called a snapshot
54:45Now the snapshot is going to be generating
54:47So when we first send a request, it's going to take a second
54:50It's going to start generating
54:51So what we need to do is we need to monitor the progress of this particular snapshot and wait until it's ready
54:58Now, as soon as the snapshot is ready, we can then download the snapshot and we can access the data, but we need to wait for it to be finished
55:05So essentially, there's these multiple API endpoints that we're going to hit
55:08The first endpoint is going to be to actually start the collection process
55:14Then we're going to use this monitor endpoint to wait for when the snapshot is ready
55:17And then as soon as it's ready, we're going to download the snapshot
55:20So we need to kind of write some code in Python here that is going to allow us to do this process where we hit the API, we wait for it to be ready, and then we download the snapshot
55:29Okay, so I'm going to start writing this out
55:34I've written it based on myself reading the documentation here, and you'll kind of see how it works as we code this out
55:40Again, we just need to make this scraper to start, and then we need to get access to the data set ID, which I'm going to show you in one second
55:46So in the left hand side here, we're going to go to discover by keyword
55:49We're going to go to the management APIs and we're going to scroll down here until we see this data set ID
55:55Okay, so we need to copy this data set ID because this is something that we're going to need to use when we actually perform this scraping operation so we can identify what data set we're talking about
56:04All right, so we have that data set ID
56:06What I'm going to do for now is I'm just going to put it in a comment in my web operations file so I don't forget it
56:12So I'm going to say data set ID is equal to that
56:16And now I'm going to start writing the Reddit search function
56:18Okay, so I'm going to say define Reddit search
56:20This is going to take in the following
56:24It's going to take in the keyword that we want to search for
56:26It's going to take in the date, which in this case I'm going to say is all time
56:30It's going to take in the sort by
56:32I'm going to sort by hot posts, but you could sort by rating or um you know up votes or whatever you want
56:38So let's go hot
56:38And then we're going to say the numbum of posts
56:43And in this case, I'm going to do 75
56:46You can do as many as you want
56:48Okay, so let's zoom out a little bit so you guys can read this and let's continue
56:52So for Reddit search, the first thing we're going to do is define our trigger URL, which is going to be the bright data API
56:57So we're going to say https/api.brightdata.com/datasv3/trigger
57:10Okay, then I'm going to say my params are equal to and I'm going to put my data set ID and this is going to be underscore ID and this is going to be equal to the ID that we had up here
57:20So I'm just going to copy it and paste it inside of here
57:24Okay, then we're going to have include errors and this is going to be true inside of quotation marks
57:34I'm going to have type and this is going to be discover new
57:39Again, I'm getting all this from the Bright Data documentation and I'm going to say discover_by and then keyword
57:45Okay, so this indicates what type of kind of search we're doing essentially
57:50Next, I need to indicate the data that we're searching for
57:53So, I'm going to say data is equal to and this is going to be a list
57:56And then inside of here, I'm going to put all the keywords that I want to search
58:01Now, you'll notice that because this is a list, I can actually put multiple sets of keywords at once and Bright Data will go and asynchronously scrape all of them for us
58:08So, what that means essentially is that if you wanted to do a 100 different search strings or a thousand different search strings, you can do that in one API request rather than having to send multiple of them because this is set up to obviously scale
58:18So, we're going to say keyword is equal to keyword
58:25We're going to say date and then this is going to be equal to the date
58:28We're going to say sort by okay and this is going to be sort by and then we're going to say the num of posts is equal to the number of posts and then again you could write this multiple times for multiple search strings
58:40So this is going to start setting it up
58:44We're now going to say raw data is equal to and I'm going to call a function here that I haven't yet defined
58:48So right now we're going to say none
58:49We're going to say if not raw data then we're going to return none
58:55Otherwise, we're going to go and we're going to parse this raw data
58:58So, I'm going to say to-do parse raw data and then we are going to return the information right here, which we'll write in 1 second
59:07Okay
59:07So, essentially this is how we're going to start uh set up the trigger
59:10But essentially, what we need to do is write a function that will allow us to download the snapshot, which is how we're going to get the data, which is what I'm going to write now
59:19So, I'm going to put another function here
59:20Here I'm going to say define underscore trigger underscore and underscore download snapshot like this
59:28I'm going to take in the trigger URL
59:30I take in the params the data and the operation name which in this case I'm just going to call operation
59:41Okay
59:41Now here what I'm going to do is I'm going to make an API request to brightite data
59:46I'm then going to get the snapshot information and I'm going to pull that snapshot until it's ready and then download it
59:53This is going to be a little bit of code, so just bear with me here
59:54I'm gonna say trigger result is equal to underscoreake API request
59:58So the function that we wrote before, we're going to pass our trigger URL, our params, which is equal to the parameters, and our JSON, which is equal to the data
01:00:09And then what we're going to do is say if not trigger result then return none because of course if it didn't give us anything then we can't pull it
01:00:18Otherwise we're going to say the snapshot and this is going to be underscore ID is equal to the trigger result.get snapshot ID or snapshot ID sorry
01:00:28We're going to say if not snapshot ID then same thing return none because we don't have any snapshot to retrieve
01:00:38Otherwise, what we're going to do here, I'm going to write a to-do, is pull the snapshot
01:00:44Okay, so that's what we need to do
01:00:44Now, we need to write some more functions uh to essentially pull the snapshot and download the snapshot
01:00:51Now, to make this a little bit cleaner, I'm going to make a new file here
01:00:54For this new file, I'm going to call this the snapshot_operations
01:01:01py
01:01:01And I'm just going to copy in this file just to save us a little bit of time because it's pretty kind of redundant code and it's not super valuable for you to write all of this manually
01:01:11So what I'm going to do is paste it in
01:01:12It's about 70 lines
01:01:12I'm going to walk through exactly what it's doing, but you can just simply download this code by going to the link in the description for the GitHub repository, finding this file, just copying it, and pasting it in here
01:01:23Anyways, let me walk through what we're doing here
01:01:25So you can see we're importing OS time requests and typing
01:01:30What I'm doing is I'm pulling the snapshot status
01:01:32So, I'm getting my Bright Data API key
01:01:34I'm setting up the progress URL
01:01:37I'm setting up my headers
01:01:39And what I'm saying here is, okay, I want to keep sending requests to this endpoint until eventually it tells me that it's ready
01:01:46So, I'm going to do this a maximum of 60 times
01:01:48I'm going to delay by 5 seconds in between each of those so that this takes me a maximum of 5 minutes
01:01:52I'm going to say, okay, checking snapshot progress
01:01:56This is the attempt
01:01:56We're going to get the response from this URL
01:02:00We're going to check the status
01:02:00If it's ready, we return true
01:02:02If it's failed, we return false
01:02:05If it's still running, then we just add a time delay
01:02:07And we keep doing this
01:02:09Okay? And we keep going and we keep going and we keep going until eventually it fails or it's ready
01:02:15Now, we have another function called download snapshot
01:02:16And we only call this function once the snapshot is ready
01:02:19So, same thing
01:02:21We set up our API key and our download URL
01:02:22And then we simply send a request where it downloads a snapshot and then returns the data to us
01:02:28Okay
01:02:28Okay, so that's all that I put inside of this function
01:02:30So now from web operations, we're going to import those functions
01:02:34So we're going to say from and this going to be snapshot operations import and then we are going to import the download snapshot and the pull snapshot status
01:02:44Okay, so now let's go to our to-dos
01:02:46So we have a to-do here where we need to pull the snapshot
01:02:48So for pulling the snapshot, we're going to do the following
01:02:52We are going to say if not pull snapshot status and the snapshot ID then we are going to return none
01:03:01What this is going to do is it's going to continually pull the snapshot until it eventually gets a result of true or false
01:03:06True means we can download it
01:03:08False means there's an error in which case we return none
01:03:09So we're now going to say the raw data is equal to the download snapshot and we're going to download the snapshot with this snapshot ID which will contain our scraped data and then we can return raw data
01:03:26Okay, so this function trigger and download snapshot is going to well do that
01:03:30Okay, so now we can go to Reddit search
01:03:32Let me just add a new line here
01:03:35From this we can get the raw data now
01:03:38So we can say the raw data is equal to underscore trigger and download snapshot
01:03:42We are going to pass the trigger URL
01:03:45We're going to pass the params our data and an operation name which I'll just call Reddit in case we want to do some logging later on
01:03:51Same thing if there's no raw data we'll return none
01:03:55Otherwise we're going to parse this data
01:03:57So I'm going to say parsed data is equal to an empty list
01:04:03that what I want to do is I want to take all of the data that was returned to us and I just want to get the information from this data that I care about
01:04:08That's because I don't want to pass all this unnecessary data to my LLM when I start checking which post we actually want to download or want to get the information from
01:04:19So I just want to get for example the description of the post and the title of the post or the title of the post and the URL of the post just the data that I actually need
01:04:26Okay
01:04:26So I'm going to say parse data is equal to a list
01:04:31I'm going to say for post in raw data
01:04:35Then I'm going to say the parsed post is equal to and I'm going to say my title is equal to the post.get and then title and I'm going to say my URL is equal to the post.get and then URL
01:04:50Now each post in my raw data is going to have a ton of information, right? It's going to have the number of likes, number of upvotes, the number of comments
01:04:58It's going to have a description
01:05:00and it's going to have the date was posted
01:05:01It's going to have the author
01:05:02I don't care about all that information
01:05:04So, I'm just parsing through it, getting the information I do care about
01:05:07And then I'm going to say parse data.append
01:05:09And I'm going to add this post to that data
01:05:10Then here I can return my parsed data like that
01:05:16Okay
01:05:16So, this function now should actually work where if we do Reddit search, it should essentially trigger this scrape operation to start happening
01:05:26So, Bright Data will go to Reddit, it will do the search, and it will start collecting all of the relevant posts
01:05:30Then, we're going to pull that snapshot because it takes a second to run
01:05:33As soon as the snapshot is ready, we're going to download the snapshot
01:05:36We're going to parse through the results and then we're going to return that parsed data
01:05:40Now, the next step after this would then be to get the URLs from this parsed data that we want to explore further and then to download all of their comments
01:05:48So, we're going to do another operation in a second here that's going to get all of the comments from a list of posts
01:05:53But for now, let's test this one out by going back to main.py and actually calling this function now from our Reddit search
01:06:01Okay, so we're going to go to Reddit search now
01:06:02And we're just going to change this to call the Reddit search function
01:06:05So what did we call this? Actually, we called this Reddit search
01:06:10And actually, let's call this Reddit search API
01:06:12Uh because if we name it the same thing as our function here, that's going to be an error
01:06:16So we're going to say Reddit search API like that
01:06:17And then we're going to import this
01:06:20So, let's go up here and let's import the Reddit search API
01:06:26Okay, cool
01:06:26Come back here and same thing
01:06:29We'll just pass the user's question
01:06:31And then what we can do is we can print out the Reddit results
01:06:34Okay, so now we've tested that function
01:06:37So, let's run this and see if it works
01:06:40And we're going to say, should I buy AMD stock? Okay, and it says it's starting the search
01:06:47And it gave us a bad request for this URL
01:06:50Also, I probably just typed something incorrectly and I will check what that problem is
01:06:54Okay, I was just checking here in kind of a silly mistake, but I accidentally had a capital T when I typed all time here and this needs to be a lowercase T
01:07:02Uh, that should fix the problem for us now in this function
01:07:07So, if we come here, we can run this and we can say, you know, invest in AMD and then we should be good
01:07:16And you can see it starts checking the snapshot progress
01:07:18Okay
01:07:18Now, while it does that, we're just going to make a small change to the code as well because the way that I'm returning this parsed data uh from here is not actually how I want to return it
01:07:28What I want to return instead is a format that makes a little bit more sense
01:07:32So, I'm going to say return
01:07:34So, I'm going to put a set of braces and I'm going to say parsed posts and this is going to be equal to the parsed data
01:07:39And then I'm going to say the total found
01:07:44Okay, total found is equal to the len of the parsed data
01:07:49Okay, cool
01:07:49So that's that
01:07:53And if we go here, it looks like it finished running
01:07:55And you can see that we get some posts and some titles from Reddit
01:07:58Now, these don't seem to make a ton of sense to me
01:08:00So I'm just going to quit this and try again because I think I may have messed something up in the search string here specifically because I think I spelled invest incorrectly
01:08:10But let's just search Nvidia here
01:08:12And let's see if we get some posts that make a little bit more sense here from Reddit
01:08:16Okay
01:08:16And there we go
01:08:19So, these make a lot more sense, right? Nvidia is actually in the post title because I didn't spell it incorrectly this time
01:08:24And there's 75 posts as we go through here
01:08:26And the next step is going to be to narrow those down so we can grab all of the comments that we need from them
01:08:31Okay, so that is working
01:08:34And we've got this first function where we're doing the Reddit search
01:08:38Next, we want to get all of the Reddit posts
01:08:40So, what I'm going to do is write another function here
01:08:42And then this will wrap up all of the search operations, and we'll go back into Langraph and start doing some of the kind of prompting with the LLM
01:08:49I just like to get the data first
01:08:51Then once we have the data, we can pass it to the LLM, and we can kind of analyze it
01:08:55So, here we're going to go Reddit_post retrieval
01:09:04Okay? And we should spell retrieval correctly
01:09:07What we're going to do is we're going to take a list of URLs
01:09:10We're going to say days back
01:09:10So, this is the number of comments that we want to get or how many days back we want to get the comments from
01:09:15We're going to say load all replies
01:09:16For now, this is going to be equal to false
01:09:19But if you wanted to get all of the nested replies, then you could go with true
01:09:23And we're going to say comment limit
01:09:25And for now, we'll just make this an empty string
01:09:29And then later, we can add a limit if we're getting too many results
01:09:31And what we're going to do is we're going to say if not URLs, then we're just going to return none because if you don't pass me any URLs, well then there's no reason to do this search
01:09:39And then what we need to do is set up a similar thing to before
01:09:44So I'm going to copy this trigger URL
01:09:47And I'm going to put this here and we're going to say the trigger URL is equal to the following
01:09:51Then we're going to say the params are equal to and we're going to say the data set ID
01:09:55The data set ID here is going to be different
01:09:59I'll show you where to get that from in a second
01:10:03And we're going to say include underscore errors is true
01:10:07Again, we're then going to create our data
01:10:10So, we're going to say data is equal to same thing
01:10:11You could run this at scale if you want
01:10:13And we're going to say URL is equal to URL
01:10:15We're going to say the days back is days back
01:10:22We're going to say load all replies
01:10:24Load all dollar replies
01:10:24And we're going to say comment limit is equal to comment limit
01:10:27And then this is going to be for URL in URLs
01:10:30So essentially we're creating one of these entries for every single URL, passing that all inside of here, and then we'll get all of the comments for all of these URLs with these parameters
01:10:41Okay, so now we need to find the data set ID
01:10:43So what we're going to do is go back to write data here
01:10:48We're going to go to where it says collect by URL
01:10:50We're going to go to the management API
01:10:53And then if you scroll here, you'll see this new data set ID, which is the one that we're going to copy
01:10:58We'll come back here and we'll paste that updated data set
01:11:00So this is the one for getting the comments
01:11:01All right
01:11:03Now, we're effectively going to do the same thing that we did before
01:11:04So here we're going to say the raw data is equal to and it's going to be underscore trigger and download snapshot
01:11:13We're going to pass the trigger URL
01:11:15We're going to pass our params data and the operation name is going to be Reddit comments
01:11:20Okay, we're going to say if not raw data, then we're going to return none
01:11:26And then if we do have raw data, we are going to parse the comments
01:11:29So we're going to say parsed comments is equal to an empty list
01:11:34We're going to say for comment in the raw data
01:11:40We're going to say the parsed comment is equal to and then we're going to start writing the comments
01:11:44We're going to say the comment ID is equal to comment.get
01:11:49And then this is going to be the comment id
01:11:52We're going to say the content is equal to the comment.get and this is going to be the content
01:12:02We're going to say the date and this is going to be the comment.get and then of course the date
01:12:06We're going to say the parent comment ID because this will be important for the linkage
01:12:10So parent comment ID is equal to comment.get get and then this is the parent comment ID
01:12:22Okay
01:12:22And then lastly, we're going to say the postc_title and this is going to be the comment.get post title
01:12:28And make sure we don't forget to put that inside of quotes
01:12:31Okay
01:12:31Then we're going to say the parsed comments.append the parsed comment
01:12:34And then lastly from here we're going to say return and we're going to return the comments which is the parsed comments
01:12:46and we're going to say the total underscore retrieved
01:12:52Okay
01:12:52And this is going to be equal to the len of the parsed comments
01:12:54Okay
01:12:54So that's it for getting the comments
01:12:59Now again, it's literally the exact same thing as getting the post except we're changing the data set ID and a few of the different parameters
01:13:08That's it
01:13:08So now we have the ability to get all of the different uh comments for a particular post
01:13:14Now, before we can test this, we need to know what posts we want to get the comments for
01:13:18So, what we're going to do is close out of this
01:13:21We're going to close out a snapshot
01:13:22We're going to close out av
01:13:23And now, we're just going to be working inside of this main file
01:13:27So, the Reddit search works, the Bing search works, and the Google search works
01:13:31Now, the next step is to analyze the Reddit search, pull out the relevant URLs, right? So, that's what this is doing right here
01:13:38And then to retrieve those comments or really the post data from those particular posts
01:13:44So let's move on and let's handle that
01:13:46All right, so let's move on to the next step here where we are going to analyze the Reddit post and then pull out the ones that are relevant
01:13:51Now, in order to do that, we're going to need some LLM operations here
01:13:55And I'm going to make a new file and I'm going to call this prompts
01:13:59py
01:13:59Now, similarly to before, I'm not going to write all of this out from scratch because it is a good amount of code and it's not super valuable to do that
01:14:07But what I'm going to do is paste in all of the prompts that I've already written that we're going to use for this video
01:14:12Again, you can get these from the link in the description
01:14:15Just go to the GitHub repository and download them
01:14:18So, I'm going to paste it in
01:14:18It's going to look like a lot of code, but really most of it is just prompts that I've already written that I've tested that work well here
01:14:25Now, you see that I have this class called prompt template
01:14:30I have a few static methods inside of here where when you call this function, essentially just returns to you the prompt
01:14:35So, for example, the Reddit URL analysis system
01:14:37That's the one we're about to use
01:14:38You're an expert at analyzing social media content
01:14:40Your task is to examine Reddit search results and identify the most relevant post that would provide valuable additional information
01:14:46You get the idea
01:14:46Okay
01:14:46And then we tell it do the following
01:14:48You know, find this information blah blah blah
01:14:52And then return a structured response with the selected URL
01:14:54Sorry
01:14:56Reddit URL analysis user user prompt for analyzing Reddit URLs
01:14:59So same thing user question that we pass the user question here
01:15:03Pass the Reddit results analyze these Reddit results
01:15:05Same thing for the Google Analysis
01:15:07Okay
01:15:07Pull this in
01:15:09Google analysis user
01:15:09same thing the user prompt
01:15:11So we have the system prompt and the user prompt and all these functions or methods that contain the prompts and allow us to pass some variables and have it kind of embedded inside of here
01:15:20So don't worry too much about this uh but there's a few functions that of course we are going to use from this file
01:15:25So now we're going to go to main.py and we are going to import them
01:15:30So we're going to say from prompts and we're essentially just going to import all of the functions that we have written there
01:15:37So what this is going to be is get the underscore Reddit analysis messages, get the underscore Google analysis messages, get the underscore Bing analysis messages, get the Reddit analysis messages, and get the synthesis message
01:15:56Okay? And then we can format this
01:15:59So we can just put set of parenthesis here
01:16:01Okay? So like that
01:16:03And then we can move this down to the next line and kind of put all of them like this
01:16:06So they're all getting imported from the same place
01:16:07Okay
01:16:07So I think that is good
01:16:11We have the prompts
01:16:16Now what we're going to do is go over to analyze Reddit posts and we're going to start using some of these prompts when we call the LLM
01:16:22Okay
01:16:22So first things first, we're going to get the user question
01:16:27So we're going to say user question is state.get user question and we're going to get the Reddit results
01:16:32So, we're going to say the Reddit results is equal to state.get and then you guessed it, this is going to be the Reddit results or an empty string
01:16:41Okay, so we're going to say if not Reddit results
01:16:43So, for some reason we don't have any, which can happen, then we're going to say return and then we're just going to return the selected URLs equal to an empty list because we won't have any to select
01:16:54Next, we're going to say structured_Lm is equal to llmwith_structured output
01:17:01And what I'm going to pass here is something called a pidantic model, which will force the LLM to give me an output in a particular format
01:17:07So, we're going to write that now, and you're going to see how useful this actually is
01:17:12So, I'm going to make a class, and this is going to be the Reddit analysis
01:17:17So we're going to say reddit URL analysis and this is going to inherit from the base model which we imported here from pi dantic
01:17:24Now what we're able to do is define a python class and then pass that to an LLM and tell the llm it needs to give us an output that's in this particular format
01:17:35This allows us to ensure that we always get something in the same format
01:17:39And in this case the format that we want is just a list of URLs
01:17:44So what we can say is selected urls
01:17:48We can say this is a list of type string and we can make this equal to a field
01:17:54This comes from pyantic and we can say the description is equal to the following
01:17:58And then I'm just going to paste in the description
01:17:59So let me copy it from my other code file here
01:18:01But essentially we just describe what we want the model to populate this field with
01:18:08So I've said this is a list of Reddit URLs that contain valuable information for answering the user's question
01:18:12So now what will happen is when I initialize the LLM I can give it this model and I can say hey you need to give me an output that's always in this format and then every time we run the LLM we're going to get selected URLs it'll be a list and it will contain the URLs that we need right that are strings
01:18:30So if we come here now we can just pass this which is the Reddit URL analysis and that's it
01:18:36We've created this structured output model and again it's just very very useful at giving us uh content in the correct format
01:18:42We're then going to say messages is equal to get Reddit analysis messages
01:18:46We're going to pass the user question and we're going to pass the Reddit results
01:18:55Okay, so these are the messages now that we're going to pass to the LLM
01:18:57And now what we're going to do is we're going to invoke the LLM and we're going to attempt to get this kind of structured output
01:19:03So, we're going to say try and we're going to set analysis is equal to the structured if we could spell structured correctly
01:19:12Not sure why it's not giving me the autocomplete
01:19:13Okay
01:19:15Dot invoke
01:19:15And then we're going to pass these list of messages
01:19:18Now, the messages if we go and look at prompts here is essentially just two messages, right? So, we have like the system prompt and then we have the user prompt
01:19:27So, that's all that we're passing
01:19:27And if we look at the Reddit one, so it's right here
01:19:30We get the Reddit analysis system prompt from the prompt template and then we get the user prompt
01:19:35So we just pass that here
01:19:37Those are the two messages
01:19:39And then what we're going to do is say the selected URLs is equal to analysis
01:19:44Okay
01:19:44Dot selected URLs because it's going to give us a Python object
01:19:50Let's fix the spelling
01:19:50Okay
01:19:50So now what we can do is we can print out the URLs just to make sure that it's actually correct
01:19:57We're getting the proper URLs
01:19:58So we can say for i, url in enumerate and we can enumerate over the selected urls
01:20:08From here what we can do is we can say print and then we can put an f
01:20:15We can put maybe a few spaces here and we can say i and then we can say dot url inside of braces
01:20:20And if we want I can just go with one here so that we start the index at one
01:20:26Okay
01:20:28Okay, so now this is just going to print out the selected URLs
01:20:29Before that, we can also just say selected URLs just so we have some logs and then we'll be able to see what those are
01:20:37Now down here, we need the accept
01:20:39So we're going to say accept exception as E
01:20:40We're going to say print and we'll just print out E
01:20:45And then we'll say selected URLs is equal to an empty list
01:20:48And then when we return the selected URLs, we'll just return the selected URLs
01:20:52Okay, so that's all that we're doing
01:20:56We essentially said all right we're going to create this structured output LLM
01:21:00What we do is we tell it that we need something in this format which we defined above
01:21:03We generate the messages that we need and then we pass that to the LLM
01:21:07So we invoke the LLM
01:21:07We get the selected URLs from the response
01:21:09We print that out
01:21:11There's some error then we print E and we say there's no selected URLs and then we keep going from there
01:21:16Okay
01:21:16So now this should actually just work
01:21:19Um what we can do is we can test this and we can see if it selects some URLs for us
01:21:23So let's go to run
01:21:25Let's run this and let's say Nvidia
01:21:29Okay, it's going to start searching for this and let's wait for the snapshot and then see what URLs end up getting selected
01:21:35Okay, so we just got an error here essentially saying that we forgot to pass one of the parameters to our functions get Reddit analysis messages
01:21:41So if we go here, you can see that we have to pass the user question, the Reddit results, and the Reddit post data
01:21:49So I think we probably are calling the wrong function uh because that's not the one we want
01:21:54We want the get Reddit URL analysis messages which just takes in two parameters and then we have another one later
01:22:02Yes
01:22:02So this one here that takes in four
01:22:05So are these called the same thing? No, they should not be get ready URL
01:22:09Yeah
01:22:09So we just misnamed this uh function call essentially
01:22:12So what we'll do is we will rename this to be get ready URL analysis messages
01:22:18That should be fixed
01:22:21And then we're going to rerun this and same thing
01:22:22Let's go invest Nvidia and see what pops up
01:22:26Okay, cool
01:22:29So, that just finished and you can see that it actually selected four URLs here and these all seem to be relevant in terms of investing
01:22:35We're investing in video
01:22:36Why are you investing in Nvidia? Nvidia is rising today
01:22:38DCA, Nvidia, Tesla, I don't know what that is
01:22:40You get the idea
01:22:43So, let's exit out of that
01:22:44That stage is completed
01:22:44And the next thing that we need to do now is we need to actually retrieve all of the comments and then get those comments and again continue to pass those to the LLM
01:22:56So we already have the function to do that, but we need to now write it inside of main.py
01:22:59So we're going to go to retrieve Reddit posts
01:23:01And what we're going to do here is just do a simple print statement and we're going to say getting Reddit post comments
01:23:09Okay, like that
01:23:13And then we can say the selected urls is equal to states.get selected reddit urls
01:23:19And then we can continue from here
01:23:21So we're going to say if not selected urls again it's possible then we're going to return and this is going to say reddit_post data
01:23:30This will be equal to an empty list
01:23:32And then what we'll do down here is we will start to collect that data
01:23:38So we're going to say print and we'll do an fstring
01:23:40We're going to say processing and then we'll say len of selected urls and we can say reddit urls like that and then we can say here the reddit post data is equal to reddit_post retrieval which we need to import
01:24:00So let's go import that from the top of our program
01:24:04So we can import that here
01:24:06Reddit post retrieval, the function that we wrote
01:24:08Scroll back down
01:24:08Okay, so Reddit post retrieval
01:24:11From here, we're just going to pass the selected URLs
01:24:15And that should be pretty much all that we need to do
01:24:16Now, down here, we're going to say if Reddit post data, then we can say print
01:24:20Now, successfully got and then we can say something like this
01:24:27Let's do an fstring successfully got len of Reddit post data posts
01:24:37Okay, let's fix the spelling here
01:24:41All right, so we successfully got those posts and then otherwise we're going to say else print failed to get post data and we can return or we can say sorry Reddit post data is equal to an empty list and then here we will go and say return the Reddit post data
01:25:00Okay, so that should retrieve the Reddit post data for us
01:25:05We're saying, okay, get the Reddit post, get the selected URLs, make sure we have some, obviously
01:25:09Uh, if we do, then we call this function, which should go and grab all of the comment data from that
01:25:14And then if we want, we can, of course, print this out
01:25:16So now we can print the Reddit post data and make sure that's working before we move into the analysis and kind of synthesize step, which will be pretty straightforward
01:25:24So let's make this bigger
01:25:26Let's run this again
01:25:26Let's go invest in Tesla
01:25:29I'm sure that's going to be all over for Reddit
01:25:32So, let's run that and let's see what we get
01:25:33Okay, so I was just doing a little bit of debugging here because the results I was getting from the comments weren't great and I realized this because I made a small mistake in the way that we are parsing this
01:25:44So, if we go back into web operations and we go to where we're parsing the comments, we need to change some of these fields because they're not actually correct based on the response that we're getting here from the API
01:25:53So the major change is that where we have content, we're going to change this to say comment
01:25:59So we're getting the comment because that's actually where the comment is stored
01:26:03Same thing for the date
01:26:05We're going to change this to date posted
01:26:07And then I think we can just remove the parent comment ID because that doesn't seem to actually be working
01:26:14And for the post title, I believe that we don't need that either because again, it wasn't populating quite a bit
01:26:21So, let's remove that and let's just stick with the comment, the content, and the date
01:26:26Again, mostly just changing this to say comment
01:26:28And then I'm going to run this again and give it another test uh to make sure we're getting the right data
01:26:32Okay
01:26:32And there we go
01:26:34It just loaded a bunch of comments for me
01:26:35And you can see now the data is actually filling in
01:26:37If we go here, there is a lot of data that we pulled because we pulled a bunch of comments from a bunch of different posts
01:26:45All right
01:26:45So, that is it for that phase
01:26:48So now we've got in the post and we've got the comments from the post
01:26:51The next step is really to synthesize all of this data together, which is going to be pretty straightforward
01:26:56We just need to write these four functions
01:26:57So let's get started here with our analyze Google results
01:27:02And then we can just go through the rest of them
01:27:03Again, it's pretty much going to be copy and paste uh but just changing a few things and changing kind of the prompt that we're using
01:27:08So we're going to do a print statement here
01:27:11We're going to say print analyzing Google search results like that
01:27:15We're going to get the user's question
01:27:18So, we're going to say user question state.get user question
01:27:20We're going to say Google results is equal to state.get and then get the Google results
01:27:27Okay
01:27:29Then from here, we're going to say messages is equal to get Google analysis messages
01:27:36And then we're going to pass the user question and the Google results
01:27:39We're then going to say the reply is equal to llm.invoke
01:27:45and we're just going to invoke the messages and we're going to go here and we're going to say the Google analysis is equal to the reply content
01:27:51All right, so the llm is just the one that we defined right at the beginning, right? So if we go here, the chat model
01:28:00So we're just calling it raw without doing anything else and essentially just getting whatever response it has based on our prompt
01:28:05Again, you can go read the prompt from in here, but essentially we're just creating a prompt that says, hey, you know, analyze these Google results and give us the interesting uh output
01:28:14Okay, so let's copy the exact same thing for Bing, except we're just going to change everything to say Bing essentially
01:28:22So rather than the Google results, this is going to be the Bing
01:28:26Change this to Bing
01:28:26Same thing
01:28:26This is going to be Bing
01:28:31Okay
01:28:31And then this needs to be Bing as well
01:28:33for the return
01:28:33We can change the print statement as well
01:28:36Okay, cool
01:28:36So, that's pretty much it there
01:28:40And rather than get Google, this is going to be get Bing
01:28:44All right, so let's copy the same thing and we're going to do it again
01:28:49This time for Reddit
01:28:49Okay
01:28:49Now, for Reddit, it's actually going to look a little bit different
01:28:53So, we'll modify this a bit more
01:28:55So, let's paste this in here
01:28:57I'm going to say analyze Reddit search results
01:29:00Rather than just getting the Bing results, we're going to say Reddit results, this is going to be Reddit results
01:29:05But then not just the Reddit results, we also need to get the Reddit post data
01:29:10So we're going to say Reddit_post data
01:29:13It's equal to Reddit_ost data
01:29:17For the messages, this is going to be get Reddit analysis messages
01:29:19This takes in three things
01:29:22So the results, the post data, and the user question
01:29:24So let's go to Reddit results and then change this to say Reddit post data like that
01:29:32Okay
01:29:32And then same thing
01:29:32This is just going to say Reddit like that
01:29:37Let's make sure everything else has changed
01:29:38So analyzing Reddit results
01:29:40Okay
01:29:40Get the user question Reddit results
01:29:41Reddit post data
01:29:41Invoke the LLM
01:29:44And then there we go
01:29:44Okay
01:29:44And then the last thing that we need to do is synthesize all of our analysis
01:29:47This is going to be quite a bit different
01:29:49So we'll just write this manually
01:29:50We're going to say print combine all results together
01:29:57We're going to say the user question is state.get user question
01:30:01We're going to get the Google analysis first
01:30:04So state.get Google analysis
01:30:07We're going to say the Bing analysis is the state.get Bing analysis
01:30:09And then the Reddit analysis is going to be the same thing for the Reddit analysis
01:30:15We're going to say messages is equal to get the synthesis messages
01:30:17And we're going to pass the user question, the Google analysis, the Bing analysis, and the Reddit analysis
01:30:23We're then going to say the reply is equal to llm.invoke the messages
01:30:30We're going to say the final answer, okay, is equal to the reply.content
01:30:41And then what we're going to do is we're going to pass the final answer, which will be the final answer
01:30:44We also need to pass messages because this is kind of how langlow works
01:30:48And we're going to pass this where we say roll and this is going to be assistant
01:30:53Okay
01:30:53And then we're going to say content is the final answer
01:30:59All right
01:30:59And let me zoom out a little bit and kind of close this sidebar so you guys can see what's going on
01:31:04Let's close the terminal as well
01:31:04So again, what we've done here is we said, okay, we're going to get all the results that we analyzed previously, right? We're going to combine that into a message, pass that to the LLM again, and then it's going to synthesize all of that together and return to us a final answer and also just a final message
01:31:17We need this message again for the lang flow kind of chain to operate properly
01:31:24So that's pretty much it
01:31:24I mean, I know that's a lot of code and we went through a bunch of stuff in this video
01:31:27Again, all the code will be available from the link in the description, but of course, we need to test this and make sure it works
01:31:34So, let's bring this up and say, "Tell me if Elon Musk is a good person." Okay
01:31:40And let's go ahead and see what that tells us
01:31:44Okay
01:31:46And after a minute here, we've gone through this whole process and we get this general response here telling us what the sentiment is on if Elon Musk is a good person or not
01:31:53And if we scroll over here, we can see all of the sources where it was getting this information from, you know, Reddit comments, etc
01:32:01Okay, so pretty cool
01:32:01Now, obviously, we can make this a lot better and we can search more things and we don't just have to have one search string and we could actually have the LLM searching multiple things and giving us a really detailed response
01:32:11I just wanted to show you this to give you kind of the sense of how you create this more complex orchestration with an AI agent that's pulling in a bunch of relevant data
01:32:19In our case, our lang graph is relatively simple, right? We have the architecture that I discussed before and we have what, seven, eight nodes, something along those lines
01:32:28But if we added more nodes, we added more LM interaction, we allowed this to run a little bit longer, we can get significantly better responses
01:32:35So I think with that said, guys, that's going to wrap up this video
01:32:38If you made it to the end, give yourself a pat on the back because this is very complicated and was a long video to go through
01:32:43Again, all the code will be available from the link in the description
01:32:46Massive thanks to Bright Data for sponsoring this video and I look forward to seeing you in another one
01:32:53[Music]
