00:00In a previous video, someone asked me what the token count was
00:01It uses a lot of tokens
00:07I'm constantly trying to look for ways to incorporate AI into my software development workflows, and one of those spots that I like exploring is testing
00:15But one of the challenges is that I don't have a lot of trust in AI writing tests because sometimes I feel like it's overly confident in what it's doing
00:23So today, I wanted to show you a setup that I used with Claude Flow, using the Spark method, and we're going to see how I was able to add some baseline tests for my NuGet package called Needler
00:33Okay, to kick things off, I have my PowerShell window pulled up here
00:37This is the prompt that I used
00:41I have since gone back in my Git
00:44I'll undo that history and see what it comes up with this time, but I'm going to explain my thought process for putting this prompt together
00:47A quick note: if you're not familiar with Claude Flow, this is something that allows you to have an agentic swarm on top of Claude Code
00:55I do have a video on that if you want to check it out right here, and you can see how to set up Claude Flow for your own development process
01:03Now, you don't necessarily need Claude Flow for this; you can take the concepts here and apply them, but Claude Flow with the Spark method gives a lot more structure to what we're going to be looking at
01:13So, you can check out the Claude Flow GitHub repository
01:14They kind of walk you through what..
01:17Spark will be able to do for you, but essentially, it's going to be setting up specifications for the things that we're going to be building
01:23Without that structure, I find that AI often goes down a rabbit hole and keeps going, losing track of what it's doing, until finally it's like, "Here you go
01:35I'm done," and I'm generally pretty disappointed
01:39But with something like this, there's a lot more guidance
01:41In my NuGet package, I started putting together my projects and some sample applications where I thought, "Hey, this seems like it's doing the right thing," and a lot of the code I was putting into the package..
01:54Borrowing from another product that I've been putting together for myself, I have a lot of confidence in the code, but I don't have the unit tests and related setup in the new Nougat package
02:03So I said, "Okay, time to use Claude Flow to help out with this." For a little bit of context, this prompt is kind of big
02:08I don't like typing out prompts anymore
02:12So, I've been showing in recent videos that I’m actually using something called Whisper, and I have a foot pedal below my desk, so I can do this kind of stuff hands-free
02:20Just to quickly demonstrate, if I start speaking and typing like this, you can see that it just puts into the console exactly what I was saying
02:30So, let's get rid of that, and I’m just going to explain this
02:32Prompt
02:34One of the tricky things when you're using voice-to-text is that it's not going to be perfect, right? Like, as you're typing things, you're correcting them
02:41But generally, I'll ramble a lot and then I have to go back and patch it up
02:44But this is essentially what I came up with
02:46So I said, using XUnit version 3, and I have noticed that I really need to remind the AI tools that it's not XUnit v2, like the latest version
02:54It's actually a new NuGet package
02:57So I explicitly called out that it's not the previous v2 package
03:01And then I said that I like using QSmartMock as the mocking framework
03:06That's another package that I like to use
03:09Some people are against it, which is totally fine
03:11So I said those are the things that I want you to consider
03:14To use
03:16And then I said the test project should have a one-to-one mapping for the library
03:18In other cases, when working with LLMs, I’ve said, like, “Hey, go add tests to this.” They seem to organize tests in ways that I’m not a fan of, and that’s totally fine
03:27I just want to give it a little bit more direction
03:29So instead of one big test project trying to test all of my different libraries, I like having dedicated test projects for those libraries directly
03:39So I gave it that instruction, and then I said there should be, you know, I gave it an example — right there should be one test project for this project, and it should be named this way
03:48So, for Nexus Labs Needler plugins, there should be Nexus Labs Needler Plugins Tests, like, just..
03:56Trying to be, you know, very explicit
03:58I said do not add 'And' or 'Substitute', and do not use Fluent Assertions
04:01You can see that it wrote out 'End Substitution' because I was dictating it, but I've also noted that probably based on a lot of the training data for these large language models, they see 'End Substitution' a lot
04:13They see Fluent Assertions
04:15I don't want to use these packages; they're just not my preference
04:19But I found that even when I tell it to use Mock, it will still sometimes go ahead and add 'and' or 'substitute' as well
04:23So it's just kind of silly
04:26So again, just being very specific about what I want and do not want
04:30Then I said do not add Arrange, Act, Assert comments
04:35I don't know if you've tried using AI to write tests, but for me, with basically 100% success..
04:40Guarantee, my tests are littered with Arrange, Act, Assert
04:44And sometimes some of the tests are so trivial that there's more comment than code in the test, and I just can't stand it
04:53So I just tell it not to do that, and then I say, basically, in fact, use as few comments as possible and try to add in comments only if you find that you're needing them
05:02It's likely because the code isn't obvious
05:03So there are probably better ways to give more structure to the LLM, but this is just me talking to it like I would a person
05:12And then I try to give it guidance not to rely on mocks
05:14I am someone who writes software where the code should be mockable, so it should be such that you can use mocks and have unit tests set up
05:26That's fine
05:28But just because the code can do that for tests doesn't mean that those are always the tests I write
05:33I write code that is testable; that way, it doesn't mean those are the only tests I write
05:38And in fact, it's quite the opposite, where I do rely more on functional tests
05:43I would rather, given a more complex application, have the dependency injection container allow me to resolve types from it
05:49So I said don't rely on mocks
05:51They should be used in extreme circumstances, and I try to call that out, like for web requests—things where you're going to the internet—don't actually go do that
05:58We don't have any of that in this library right now, so it's not really necessary
06:03But I also tried to call out that if there are APIs..
06:06Boundaries for third-party projects, so for example, I have Carter in here, I have Screwdor
06:12We're not going to use them for these tests, but just trying to set the precedent that if you're going to go interact with third-party things, that's probably where I would at least consider a mock
06:21In some cases, I would just use the real thing anyway
06:25But if I don't put this in, I have found that it will over-mock things, and then it becomes kind of pointless
06:31I've literally seen it write tests on the mocks themselves, and it's not actually testing anything real
06:35So it does it with a high degree of confidence and adds zero value
06:39So, otherwise, I prefer to use real instances and resolve these instances from the dependency containers
06:43Necessary
06:46In these particular tests, it's not really going to be necessary
06:50But, for example, if I had a service in my application, I would ask it to go make the dependency container the same way that the application does and resolve that service
06:59So it has the real things inside of it, not mocks of everything
07:04Or you'll sometimes see test setup code where it's duplicating the construction of an object, and there might be a lot of dependencies, and it's just a mess because if that code changes in the actual application, we're now testing things set up in a different way
07:20I just try to avoid that, and then I kind of give it that guidance—like, if you're just testing the dedicated library and it doesn't rely on a Just instantiate the objects, giving it a bit of an outline
07:32Then I said I just wanted to focus on this particular project
07:34So, this one is Nexus Labs Needle Injection
07:38I specified to focus on a couple of folders, such as Loaders, Sorters, and Type Filterers
07:44I felt this provided enough context because there are several classes in those folders that are related, and I didn't ask it to write tests for everything
07:54I find that when you go too broad, the process falls apart much faster
07:59You might get a few tests where you think, "This feels okay," but beyond that, the results can be really poor
08:01So, this is the prompt I used
08:08Let's see if it does what we need it to do
08:10While that's happening, I will poke around
08:12around a little bit in the solution to show you where it's going to hopefully add some tests
08:16So, let's go run it
08:24Cool! So, you can see that it's getting everything up and running, and it's just going to use the Spark method and pass my prompt inside
08:29So, while that's running, I'll come back to it
08:32But just to show you, we are going into this project
08:36You can see that I have loaders, sorters, and filterers
08:38So, basically, we should be seeing tests generated for these three, these four, and these two classes
08:46We'll see what it does
08:49These are all pretty lightweight things; there's not a ton of really complex logic
08:53This should be something that's quite simple to test
08:57It's just nice in this case that I can have AI, hopefully, go
08:59It does most of that work for me
09:02But we'll see how good it does because I suspect we'll still need to go through it a little bit and give it some TLC so that it's acceptable
09:09Right, we are done
09:16And I just wanted to call out that in a previous video, someone asked me what the token count was
09:24I glanced up a couple of times while this was running, and I saw that the tokens were resetting
09:28It uses a lot of tokens
09:32I was seeing some counts get up to like 15,000, and then it would go off and do something and come back, and I would see the count go back up again to about 15,000 or so
09:40So, I don't know exactly how many tokens it uses, but it's an awful lot
09:44Let's see what it says it did, and then we'll go through some of the tests
09:47It added XUnit v3 and Moq
09:49I saw this
09:51Happen pretty early
09:54We are using central package management, so I noted that it was putting stuff into here
09:59We'll confirm that, though
10:04It went and added test projects for all of the different projects I have in the solution, which I don't know if that's exactly what I meant by my prompt, but this is the second time I run that prompt and both times it did that
10:09So probably something I need to be more explicit about
10:13Totally fine because I think they're empty except for the one that I asked it to add tests for, which is Part Three
10:19I implemented comprehensive tests for Nexus Labs Needle Injector, including loaders, sorters, and type filters
10:24It looks like, if you recall what I said before, it tackled the right classes there
10:28That's good news
10:31And then it says test characteristics following your requirements: XUnit v3, not v2
10:34Mock only where necessary
10:37No end substitute
10:39No fluent assertions
10:41We'll check to see if it added comments or not
10:42Uh, minimal comments only where absolutely necessary
10:48Real instances use actual implementations
10:50It says 64 tests, 59 in injection, and then it added five in Needleman tests
10:55So, interestingly, I didn't ask it to do that
10:58I did glance up partway through the run, and I saw that it was doing that
11:03The reason, just for context, is that it said that it was adding five there to prove that that project was building
11:10Kind of odd, but that's not what I asked it to do, and it still went ahead and did it
11:14So, 64 tests total
11:17Then, 59 for what I told it to do
11:19So, let's jump over to the code
11:22The solution has to be reloaded because it's changed it
11:26So, um, okay
11:30So, something I see that I don't like already is, yes, it did put, you know, 'test' at the end of the project name for all of the different projects
11:32That's great
11:34That's what I wanted
11:36I personally don't like just blasting the tests out into their own subfolder, but I wasn't transparent or explicit about that request, right? It just did what it thinks is the right thing to do
11:50So, that's not the end of the world, but not what I would like
11:53So, generally, I would go move these things out of the tests folder there, but that's fine
11:57This is where I asked it to go add tests, and we can see that it did
12:01Add those test files
12:02It might not have been obvious because the prompt is probably flying by when it was doing all of the work, so, the console—I mean, but it did say that it was running the tests and they should pass
12:16We'll verify that
12:29But what I wanted to do is step through these tests and see if they actually make sense, because one of the biggest sort of risks or concerns I have with AI writing tests is that you might have noticed this, but when AI gives you answers, it sounds like it's very confident, right? So even when I was going through that checklist, the output was very much like, yep, got all this stuff delivered exactly as you wanted, checks the box, but like, does it—did it write tests, but are they actually useful? Tests? I have this class called the All Assemblies Loader
12:43And if we were to look into it, we're not going to do this for everyone specifically, but it's going to try and look for everything with an executable or DLL in the base directory
12:53So, where we're running from
12:55Not really that complicated
12:57It tries to load all assemblies from the base directory
13:00Does this test actually do what it says? The answer is no, which is kind of funny
13:07It says the expectation is that it loads all assemblies from the base directory
13:11But if we look at the assertions, it's just checking that the return type is not null and not empty
13:17How many assemblies are in the base directory? Is it 100? Is it one? I don't know
13:24Right? Like this, the test does not actually confirm that
13:26So, this one isn't that the test itself is bad; it's just that it's misleading
13:33The expectation that it's putting into the title is not actually aligned with the assertion
13:38So, that's not a great example, unfortunately
13:41Okay, next one
13:43Contains current test assembly
13:45Does this one actually do that? It looks like the answer is yes
13:48It's checking what the current assembly is
13:51If you're not familiar with this stuff, Assembly.GetExecutingAssembly() is going to give you the currently executing assembly, as the name suggests
13:57So, then we're looking through that assembly's collection and making sure that we have a name that matches
14:02So, I think that's pretty
14:06Good
14:11You could maybe use the location of that assembly instead of the name, but that test seems like it makes sense, right? If we're asking for all of the assemblies when we call this thing, it should contain the current one
14:15I think that's an okay test
14:18This one is saying that it loads DLLs and executable files, and then it's doing an assert across all of the results that they must end in DLL or executable, like .dll or .exe
14:30So I think that's a totally fine test to create
14:32This one says that with "continue on errors" set to true, it handles invalid files
14:37This is interesting because when I look at this test and what it’s trying to do, for this loader I basically have a condition that says if you encounter an error, just continue, but..
14:45That's optional
14:53It's testing with that option on, so we should be able to continue, and that means this should pass, right? And so, that's fair
15:00So far, so good
15:07But do we know that this is going to throw an error? Like, when it goes over the assemblies in my output folder, is that going to throw an error? Maybe not
15:14So, we don't actually know from this test if it's handling the error
15:17All that this test tells us is that whatever it's running against doesn't throw an error, but we don't know if it's actually going to simulate that condition
15:27That's a bit of a risk
15:29It's misleading in terms of confidence because it's telling us we have a test that guarantees this; it doesn't guarantee it
15:36The other thing is that we don't..
15:40I have the inverse
15:44And that would be something that's really helpful here is that if we had the inverse, where "continue on assembly error" is false, then we could explicitly have a scenario that did the opposite
15:49That would be helpful to show us that, given these two scenarios, we can have the error either bubble up, get thrown, or have the error continue over
15:58So, so far, this isn't terrible, but it’s just reassuring to me that I will not stop reviewing AI-generated tests
16:07We're not going to go through all of these in extreme detail
16:11I figured I'd jump over to another one
16:13Let's go over to this default type filter one
16:15So, again, these tests are all quite simple, right? If I scroll through pretty quickly, you can see
16:22That's like, you know, one line of action and then a check, right? Like an assertion
16:26And so what this one is doing, this class is the default type filter
16:30It's looking to see whether or not, because this is a dependency injection framework, it's checking to see if we want to have types registered by default as transient or singleton
16:44And so it's just testing that expected behavior
16:46What it did was generate a bunch of different types for us, right? So all the things where you see the `typeof` keyword here, these are all examples that it was coming up with
16:57If I just jump to one of these, you can see that there's a whole pile of them listed at the bottom
17:02And what's interesting is that there's a bunch of them inside this
17:07class
17:09And technically, none of those should get registered by default because they're inside of a type
17:13I know the implementation because I wrote the framework
17:16What it should do is check that none of those actually come up
17:18That's what it should do
17:20It's hard to know
17:22We'd have to go through these one by one to see if it missed an edge case
17:26So again, this is a kind of situation where I would really urge you that if it spits out a whole bunch of different test scenarios, you definitely want to review them because just having a large number of tests, like what was it, around 60, doesn't necessarily mean they're good tests
17:45They could be very misleading
17:47Something else to point out, just to quickly scan
17:49through here
17:51There's probably a lot of these tests, and if you notice, they're almost the exact same structure, right? The injectable type, is injectable type, assert false, assert true, whatever
18:00We could probably use an XUnit theory here
18:02So some of the things I'm thinking of are like, I would go back to Claude and say, "Great, like, for the default type filterer tests, refactor this to use XUnit theories instead," and we'd see a lot of this code get collapsed down
18:10Not terrible
18:17This one probably gives me some confidence, but I would need to go through and make sure that it's testing the things I care about
18:20Maybe let's look at one more
18:23This is an assembly sorter
18:26Let's do the alphabetical one, because alphabetical is pretty
18:31Obvious, I think this is going to check that the assemblies it's loading are in alphabetical order, which makes sense
18:36If we're sorting with an empty list, so these are some edge cases
18:40If we pass it a null, it should throw because there are guard clauses in there
18:43This feels pretty good to me
18:46It sorts with a single assembly, and we get one back
18:49If we have duplicate names, what should it do? So, I don't have any error handling, like if there are two with the same name, should it throw? It doesn't do that
18:57This is a totally valid test
18:59And then it's giving us a list of assemblies to work with, but this isn't a terrible one
19:04This is probably pretty decent
19:06Like I said, a lot of these classes are very simple, so I'm just trying to..
19:10I'll give you some thoughts for when I'm reviewing AI-generated code, especially with tests, like what I'm looking for
19:15But if we go over to the test explorer, that's popping out way too far
19:17That's okay
19:19I'm going to go run them all, and we'll see
19:21It told me in the console output that it did run them and they all passed
19:24So, if anything fails, I would be very confused, but it's possible
19:30And there we go
19:31Right
19:32So, it does actually fail when it's running here
19:34Alphabetical, and this one here says it fails
19:37And what does it fail on? It says it's not in alphabetical order because it's looking for these things out of order
19:44Um, it's just cut off because of the resolution of my screen
19:49So, interesting fact that again, you don't have this
19:54Because you didn't see it, but I have run this prompt before, and it generated tests, and I've already gone through this manually, just that it made different tests
20:01But this is actually a test that it recreated
20:04This is the second time I've seen it
20:06I think what's happening is that when it's running the assembly loading, I believe there's a difference between running it on the command line versus here
20:13I think that this likely has to do with the fact that on the command line, it's running in a WSL environment, so it's a Linux Windows Subsystem for Linux, compared to me running it in Visual Studio
20:23That's something that, when I was dealing with this the first time, I pushed up to my build system, and that broke
20:29Because I think it's also running on Linux there, something to consider
20:33But there's one more thing, and I was hoping I would easily come across an example here
20:39There's one more thing that I think is kind of dangerous that it does in these tests, and I'm just trying to scan through
20:43Sorry
20:47I saw somewhere it said it was going to do it, but I don't see it coming up
20:48Maybe it's in here
20:50Okay, maybe not
20:52I can't really see an example of it
20:54But just to explain something that I've seen to be kind of dangerous is that there's a rule that you're not supposed to have if statements in your tests
21:02And I don't like saying, you know, like always or never
21:05If you've heard me talk about this kind of stuff before, I think there's always Exceptions
21:09But in this case, I have seen it basically in a single fact, which is not a parameterized test, right? None of these tests are parameterized
21:17It will have an if statement here around an assertion because it's like, oh, sometimes this doesn't pass
21:23So, it puts an if statement around it
21:26To me, this is extremely dangerous because it means that you have some test code that is sometimes not running
21:32If there's an if statement around it, that means that sometimes that condition will look different
21:38I would really encourage you, when you're reviewing AI-generated tests, to look for if statements
21:43If you're writing parameterized tests and you have if statements and things like that, it's a bit of a smell
21:49I'm not going to, you know, I wouldn't freak out necessarily, but it's just something you want to be careful about when you have if statements in your tests
21:56But in fact, for a single test, not parameterized, I really think that having an if statement in there is going to be dangerous
22:03So I've noticed that when it is running tests, looking at the output and trying to refine it, sometimes instead of just fixing the faulty test or if there's something wrong in the code, it's like, hm, if I put an if statement in the test, it works around it
22:18So be very, very careful about that
22:21In general, though, I'm not very upset with this
22:23I think it did a decent job, but I would personally be going through all of these and, you know, double-checking
22:29Them, because it wrote 60 or so tests
22:33That's a bunch of code to review
22:36I don't recommend blindly just taking that, pushing it, and saying, "Hey, look, green check marks
22:40Everyone's happy." I would go through them one by one and make sure that the tests make sense
22:44One more quick note is that if the format of your tests that it creates is not something you like, either ask the LLM to go change that, so go back to Claude and say, "Hey, I want you to do it this way
22:58Structure them differently," like the example I gave you with using facts versus theories
23:01If you like having more parameterized tests, encourage it to do that
23:06The reason I say that is if you start landing code into your codebase that's following patterns… That you don't want to have propagated, and you keep using AI, it will keep propagating these crappy patterns that you don't want to have
23:12So, just a friendly reminder
23:18But in general, I hope you found that interesting
23:25It gives you some ideas to think about when you're using AI to write your tests for you
23:28And yes, this is a NuGet package that I will be releasing
23:32It is in very early alpha stages
23:35It's called Needler, and the goal of it is to provide an opinionated dependency injection setup that allows you to scan assemblies and automatically register types
23:44I use this in my own applications, like BrandGhost, that I build on the side
23:49So, thank you so much for watching, and I will see you in the next one
23:52Take care.
