Course Builder: Core, Adapters, Providers, and Schemas

[00:01] I am looking at Matt's Total TypeScript MonoRepo and specifically at the written content manager and the schema.prisma. This is obviously aligned with a lot of what we do in terms of our skill recordings products and the course builder products that we are working on now. As far as our own work, we are aligned more with the course builder instead of the products repo, which is what Total TypeScript is actually based on, so that's where some complications come in. For some history, I've jumped over here to the Skill Recordings products repo and kind of dug in a little bit into our packages, into the database packages, and the Prisma schema. So inside of here we can actually scroll past content resource, we'll get to that later.

[01:16] One thing that is decidedly missing from this particular repository is anything really related to content in the database. So if you scroll all through here, what we have is everything essentially related to commerce and off is what is in our Prisma. So I'm gonna come up here and I'm gonna look in total TypeScript and I'm gonna come into Schemas. And inside of Schemas is where we are actually defining the content, and that is because we are using Sanity. So these are Sanity Schemas, and we'll come in here and you'll see the concept of a module.

[02:02] I actually don't know what's going on there. It's fine. Section, tip, video resource, explainer, exercise, article, Right? Like these are all content types. And if you look at them, they have some very similar properties in terms of the data model for each kind of content.

[02:26] Modules have a type. So we have two different, well, we have a bunch of different types of modules. And essentially a module is a collection that contains sometimes other modules or interviews or explainers or exercises etc. All right so now we are looking at CourseBuilder, which is an evolution of the Skill Recordings product repository. And one of the biggest differences is we aren't using Sanity and instead have kind of embraced this idea that Course Builder is the CMS.

[03:05] And for us, that simplifies things because it removes one of the databases from the equation. We actually love Sanity. It's a great product, but for our needs, we need something that we can provide multiple people access to and found ourselves actually building Sanity UI over again in our clients anyway. So this is the answer to that. And it was kind of starting from scratch and thinking about what we've learned.

[03:30] And there's a few different packages in here that are pretty important. This is very closely modeled after Auth.js, specifically version five of Auth.js and frankly is almost a fork of that particular library. We have here the core and core is framework agnostic. It doesn't speak React. It doesn't speak Next.js.

[03:58] It is framework agnostic. And what it actually contains are schemas. And in this case, these schemas are Zod schema. So we use the term schema quite liberally, I think, across the application. These schemas in core are actually the Zod Schemas and then also the types that live next to the Schemas.

[04:21] So we have this idea of providers. And for us, these are the services that we use primarily. There's a few more services that we make use of, but If we use OpenAI, then we are going to use the OpenAI provider here and it follows an interface. That's really nice. So we can have different providers, but we are fairly strict about sticking to interfaces when we use these.

[04:46] So that's kind of nice because you can import core and then use the providers for these different services and there'll be API compatible with Course Builder and in practice, that's actually pretty cool. So inside of adapters is another interface. And to me, this is like the joy of the whole thing is programming against an interface. And if you look at the course builder adapter and what it provides, it actually extends the Skill Products Commerce SDK. So this is basically just a straight port of what we're looking at in products.

[05:27] So coming over here, looking at products, going into Prisma inside of database. And instead of looking at Prisma, I'm actually gonna look at the Prisma API and you can come in here. And this is like the API for all of the commerce and all of the stuff, right? Like get purchase details. And this is all Prisma related.

[05:46] It has a context And inside of Course Builder, we are currently using Drizzle, but what we did was export this, right? Like export the signature that we're looking for. And that is what became the Skill Products Commerce interface, which actually lives in Course Builder, but this was like, okay, I want all this stuff from Skill that already exists. And I don't want to convolute that too much with what we are adding for Course Builder, because mainly because of the new CMS functionality. So we had to, you know, like we have to extend that.

[06:22] Let me come back up to the interface. And so this is an adapter. If we look, that's actually going into AuthCore. So this depends on AuthJS in some ways. It might be nice to get away from that, but we're able to just use Auth JS here and we're using their core, which similar to the core in Course Builder is not speaking to any sort of framework and is mainly revolves around this idea of interfaces and how we can code against interfaces.

[06:52] So this course builder adapter interface is important. There's a mock one in here and stuff that gets used for tests. And then obviously that skill commerce one that we looked at below, we can see that the different types get used in here heavily because we are building an interface that's going to use those types. So the core of course builder is actually these schemas and types. And if we are building against those, we're really doing ourselves a favor and it's really nice.

[07:23] And, you know, like these are tried and true, right? Like these are what we have used and kind of put through the ringer of commerce. So a lot of the complexity in this is actually related to commerce. It would be interesting to separate this out into commerce and content and have that kind of separation. And that's something that I think would be extremely interesting if it was separated, it's really kind of hard to separate it from the idea of like a user, but that can be an interface because the user is, you know, like it's, it doesn't have an email and an ID, great.

[08:00] That's all we really need. You know, Frankly, we just really need an ID because we're doing these relations. So the course builder adapter, again, that's in packages, core, source, adapters, and that's where the primary course builder interface is. So those get implemented in adapter packages. So this is adapter drizzle, and I'll dig into that in just a second.

[08:27] The other thing that core actually does, Well, not really. So I want to come down to Next. And this is where all the, there's a little bit of Next.js pollution across a couple of these other packages, but for the most part have tried to keep them really clean and next free so that, you know, like most of the stuff related to next, I think I have commerce next. And it was just like, there was just some stuff that we were trying to get it done. And that could be cleaned up to where this was just commerce.

[08:56] That implies that this is dependent on next also. So this next package is the main one and this would be like next off. And this provides the, you know, if you come in here and start looking, that interface, right? Like that gets used as the API. So just to go up into implementation land, If we dig into any of these sites and come down here, they're all app router.

[09:22] It doesn't really matter as far as that goes. And we have this course builder route and you can see this is post and get out of course builder config. So we look in there and this is like the next auth config, right to where we have callbacks and we have different things that we can pass in like get user and stuff. So we're using, you know, like we're against these interfaces where we can pass in various overrides and callbacks and configuration, as well as delivering additional providers or concrete providers that are probably related to the ones like this is importing that from core. And then I create an OpenAI provider.

[10:07] And now I have this OpenAI interface. And in our rig that gets used in Ingest. The way we use Ingest is actually pretty cool because in core, there are ingest functions and inside of config, you can see that. So we are importing these functions. And if we look at that, I don't remember.

[10:29] It's these basically. So some of our video processing and that sort of stuff is all in core so that I don't have to repeat it and this can exist in every course builder site and give us those basic content handling things that we love in Ingest And that's really nice. And then in the actual server itself, we end up setting up middleware. So we use those providers here too. So inside of our ingest functions, we always have access to these providers as well and use those interfaces.

[11:02] So we can avoid to some extent, it's not perfect. We can avoid doing direct access to these services via HTTP or their SDKs or whatever. And instead, again, we are programming against interfaces And that's pretty cool. And it covers a pretty good gamut of the things that we generally use. And what I love is like, if we look at, I don't know, Maybe not in here.

[11:31] I'm gonna come back into Course Builder and we look in here and we have like an email list provider, right? Now look at my email list provider and it's using ConvertKit. For some sites, we don't even wanna spin up ConvertKit. So if I come into JS visualized and I look here, we have the adapter and we have an email list provider, but this is an email list provider that just uses our own database. And that's awesome because the cost is just rolled into the database And that's great.

[12:00] So again, because we are using that email list config, right? Like, so we are expecting the options to be passed in and then that config, we are against an interface and this is a drop-in replacement for ConvertKit that uses our own database. And one thing we're not doing with this that is definitely different is actually getting to the point where we're sending broadcasts and that sort of thing, but it opens up that opportunity and I think that's pretty cool. So this was kind of an experiment to actually drop in a different provider. The same would be true for an OpenAI provider if you're using Olama or some other OpenAI compatible API, you could use that here.

[12:41] Transcript provider, same thing, we're using DeepGram, but we want to be able to have that flexibility to use other providers. And in practice, that is awesome. It really, really works out great. So that's what Core does. And it gives us the kind of the last thing that Core does that I haven't talked about yet.

[13:01] It comes through Next. So inside of that API route, this is an API route handler. We see we get the get in the post, those come in. And at some point we have a net course builder. We end up in the core, right?

[13:18] Like, so we're calling, we have the course builder, right? There from the core and we initialize it. And what this basically, I ended up in a, what this basically does inside of core is passes it through, right? So we call, we make a new course builder, we pass in a request object. So this is just a normal request.

[13:47] And then we do some work on that and parse stuff out of that that we're gonna need. We actually do some stuff and grab our providers and anything that we have configured per API route. And it gives us just a really nice way to where you ping a single route on the client side on the consumption side in the apps you have that one simple route which I don't think I have here go back to that here right this is all it uses the it's from the config. So you do have this config, but that this, the route and the config is then all you technically need. And then you come in here and it has a little handler and I'm in the next package.

[14:51] So back up into core, we are going to handle that, right? Like, so we come in here and we grab all that stuff and we're inside of the core. And we get here and if it's a get, we do this stuff else, it could be anything else. Really we're expecting posts. So it should probably consider that, but these are actions, right?

[15:22] Like, so it's forward slash API, forward slash course builder, forward slash whatever the provider is, forward slash action. So you end up, might be action provider, I forget one of those two. So you end up calling that URL from the client and then it gets handled in here based on the available providers. And then this kind of action interface that we have. So, you know, like if we want to process a refund this gets called, get subscriber, formatted prices, that sort of thing.

[15:54] Most of this stuff currently you can look through here. So most of the API stuff is very focused on the either author or commerce, right? Like all of this is commerce focused. None of this is content focused. And that's why I think there's actually probably room for that separation of the idea of commerce and content, especially when you're talking about integrations, because like I'm currently working on this, right, which integrates with course builder on the egghead site.

[16:30] So it's loading data. And this is all content, I don't need, like with this, I don't need any of the commerce stuff at all. Like it just doesn't need to be involved. So there's an interface there for commerce and there's an interface for content. And I think that is something to explore more heavily.

[16:49] All right, so that's how it works. It comes in, it comes in the API route, those actions get handled, and this contains, the core contains our schemas. And where this actually comes to interface with the database. I see I have an adapter skill stack that I kind of started and haven't made much progress with, but the idea would be that for our products directory, we could just have an adapter and drop in and it would be a course builder site. Working on that, this would be Prisma too.

[17:16] Anyway, so inside of the adapter drizzle, this is really interesting to me because the adapters actually get tested. And I think that's pretty cool to look at. The tests live in, I think, where do the tests live? It's in packages. Thought it was utils.

[17:47] Oh yeah, there they are. So the tests live over here because these use the database, but they are actually database agnostic. So they don't speak Drizzle, they're just using the interfaces. So they're actually outside of the adapter Drizzle and the adapter Drizzle calls those texts. And these are mainly, again, what we focus on if you look at these tests are commerce.

[18:07] We, like that's the one thing I'm like a stickler about in terms of tests and stuff. The commerce is a huge pain in the butt and we wanna get it right. And it takes a lot of trial and error and took us a long time to get it right. So we've really focused on that. And the commerce stuff is actually a little looser.

[18:23] So coming into here, so I'll get out of the test weeds. We have a lib and inside of Auth.js, they actually have for Drizzle, there's like a Postgres and a SQLite, but I only ported them my SQL because we use PlanetScale and it works fine. But the idea is the same, right? Like you come in here and it would decide based on what flavor you're passing in, which one to use. And it has SQL flavors.

[18:52] That's pretty cool. So you can switch based on which SQL flavor you're using and then Drizzle would speak that and it would understand it at the table level. So inside of MySQL, there's all this where it's actually setting it up. And you come in here and the index is the actual adapter. And this looks very familiar.

[19:13] It's the course builder adapter. This looks very familiar to this, which is the Prisma API in our products where this was all commerce. When we end up in course builder is we have the commerce, right? Like that's all here. The adapter supports commerce, but then we have content as well.

[19:35] And I think that is where this gets interesting just because there is an API for managing content in general. So this is complex, right? Like actually implementing this takes quite a bit, though implementing it as as mocks or stubbing this out is something that we have done quite a bit. All right, so that's the adapter, but the real kind of fun, and maybe the point of this is this idea of the schemas. And you can see we have auth, we have commerce, we have communication, and we have content, and you can open those up.

[20:16] These aren't actually used very much so comments and communication channels. That's something that will come into play more as we get into sending emails and that sort of thing. But we do make use of the commerce and you can come in here and these are defining these tables. So the coupon table looks like this. This is a drizzle schema.

[20:37] It has the relationships. What is most interesting in terms of our conversation maybe though is here inside of content and specifically content resource which is the core of everything. So previously when we would look at these and what is represented here is this idea of a course, a section, an exercise, a social post, let's see, a social post collection, exercise videotape, all these things, right? And what they are in terms of Course Builder is a content resource. And what a content resource is or has is the ID.

[21:24] It has a type. So this would be section lesson, workshop, tutorial, article, whatever the content type is. And this is purely for separation, maybe at the URL level or whatever, who created it. I'm actually on the fence about this one. And then fields.

[21:45] So fields is JSON and it can be literally anything. So just a string and literally anything. We also version them. So that's something that we introduced recently. So as you create new ones, it makes a little hash and we'll save previous versions which haven't actually had that come in handy in practice, but it's something we try not for Egghead, just to see how it goes.

[22:09] And then of course, create it, update it at and delete it at we soft delete everything in these databases. And then they have all of these relationships. One of the cool things about a content resource is that a content resource actually has resources. So you can put other content resources inside of your content resource because a collection is a resource. I am going to come over here.

[22:44] All right so here on the badass site there is some details about what we have done. And I'm going to come in here to the information architecture. And there is some extremely good research from Lisa Maria Marquis. And she went through everything. And at the end of the day, what she came up with is everything is a resource.

[23:07] It's a recess resource or it's a collection of resources, but a collection is just a resource that contains other resources. So everything is a resource. And it's like naming things and being very specific about what these were at the table level is something that has plagued me forever. And we even suffer, I think, insanity though, technically speaking, this is very much built on the underlying principles of how sanity operates too. So like, this is, you know, like the idea of having a relational schema, right?

[23:46] Like, so a relational table to where I can relate content resources to whatever I want to. So, you know, like they have an owner, you know, like people can contribute to them. So you can look over here and there's different contribution types, like author or whatever, and we use those to differentiate. Here's the versions. We do tagging.

[24:08] So, you know, like we have this relationship with tags, but the join table, so content, resource, resource, and content resource are really the only two that are absolutely essential to be like compatible at the content level. And what is really neat is we take this and, you know, like we have this and there are a couple, like If you come into core and look at how we're doing this. So there is a content resource schema, which is like the basic kind of generic content resource. And then I'm like, I did video resource and I actually flattened this and I don't like it. It's missing field.

[24:50] So it's kind of weird. I don't think there's any other content stuff but where we define the actual content types is at the app level. So like if we come over here and we'll go into the soon to be renamed total node and it's not schema. Oh, sorry, lib, right? Here is where you start to see the actual content type.

[25:19] And one of the things that we have fully moved to and that I like a lot is taking this even further. And you will see like there's workshops, right? So we are defining the workshop schema and this handles stuff like navigation, which is a real chore and how does that work? And what are these all expecting and all that fun stuff. So that's there and tutorials is probably there.

[25:42] I don't know if that's the schemas or just the query. This idea of lib, this is basically SDK. And what's cool is these can actually be, you can promote these, right? Like, so these could get promoted over to core if we wanted to. But generally speaking, like from product to product, we are, you know, changing it up because a lesson and the lessons, one of them where it could probably be in core just because we use this idea, but a lesson on total node is not necessarily gonna be the same as a lesson on for like Epic React, for instance, where that, you know, like that could be totally different.

[26:16] So this is, right? So we have resources. It does not extend content resource, but which we actually do, like we merge them quite a bit to where if you merge this, let me see if I can find one where That's what's happened. Yeah, so content resource schema merge, and then we're just defining the fields, right? Like, so we're merging that into the normal content resource and then defining the fields and we can do updates and stuff.

[26:43] And post is one that we've very firmly landed on, this idea of posts as kind of the generic thing for the non-course, non-tutorial content. Though these could, like frankly, they'll have a post type in here too. So it's kind of silly because you're like nesting it down and it's like two generic things stacked in a trench coat kind of situation. But like posts are just versatile and that's what we have shifted to on egghead and and other sites. And so you know Like it's a post can have a video or not have a video.

[27:18] So it's an article or a tip or whatever it is and kind of get in a way we've talked about this a lot, getting away from the notion of the tip in particular or what do you call this thing? And it's just a post and a post can have attachments and based on those attachments, we are gonna display the post in a certain way. And that flexibility is pretty nice, but it is also nice to have something like a workshop. So we have other things like we have prompts, and prompts are just content resources, but these are used to actually write a prompt that we use inside of automation for LLM type of stuff. Let's see what else is in here.

[27:55] Module is one that's like a very familiar in terms of like our sanity schema and just defining at a granular level, the types for what a module is, either a tutorial or a workshop. And we do that just to, you know, like this is a very specific thing and these vary based on the product because how we present that product and what a module contains and how that gets presented. Like that's something that is kind of up to the creator that we're working with in terms of what they want to add to it's like StackBlitz or an external app or whatever, like we want to be able to handle that. So these are all interesting. You can come in and kind of check them out like an email, right?

[28:35] Like an email is literally just a content resource. So everything becomes a content resource and we use these fields. And in practice, the JSON blob field is awesome. Like it works really well. You have to, for these things, we separate it and we go into queries, but you can, you know, like the ability to get in there, you can index on them, like all sorts of stuff.

[28:57] And that's MySQL, it's based on which database you're using, but you know, it's really nice. And we do still end up with direct drizzle access or whatever it is you're using in the product space. You can drop down to that. So you always have that little level access, but again, like where we can, what can we do with the adapters that we have access to? And that makes it more portable.

[29:20] So as things, as we iterate and things are like, oh, this one is a good idea. Like for me, I really wanna move posts up to core because I think this idea of a post is really interesting and something that we can probably like bring some standards around in terms of how we think about posts and the kind of self-service items that any of these given websites have. And Egghead probably has one of the most interesting setups as far as that goes. And I'll come in here. It is all posts.

[29:57] And if you come into posts query, like this is set up because I'm actually building this, right? Like, so I'm building the course builder, a VS code extension. So like it has, like I've been building out API. So there's, that's API. And there's like, you know, the post route, so it's ability to manage over API in a validated way.

[30:23] It has, there's OAuth involved here. So we have all that fun stuff set up so you can OAuth and do device flows and authenticate with remote things and then you end up having to build a REST API and all that fun stuff. So that's happening. But I think what's interesting as far as that goes, This is what a post looks like here. It's a little different.

[30:48] It does have that post type. And this is really built around, you know, like there's an AKE lesson ID. So even this is a little bit of a snowflake. So the idea of how do you handle that is great because in the App Land, whatever these are, it doesn't matter because you can you can model this to what suits and and that struck me just because if we look over here like an exercise video take like yeah it has these four things that are different but you know that's a content resource that's a content resource right like it it's whatever you can you can use these as as you know like you get the point And then this idea of collection goes away because this is just another content resource, et cetera. Pretty clean in practice.

[31:31] Let's see. I thought this was also interesting. It's interesting from like an integration standpoint also, because if you look what happens here, I'm trying to find some Postgres. So Egghead runs on a Postgres database and I do a lot of syncing. So when you're creating stuff or whatever, it ends up writing it to type sense and then it also sends it over to Egghead directly.

[31:56] So I have these PG queries. So you can, we're actually writing to two databases and syncing content from Course Builder down to Egghead. Yeah, so that's it. And like what is interesting to me here, last thing up here in, we're gonna come up and we're gonna go into the database. I wanna come up into a simpler one.

[32:21] We're gonna look at Astro Party. We'll see what's in here. Inside of database, we come into the index. This is just where we get the actual database client that we can use. MySQL table, Drizzle is amazing for this because we can like ignore and prefix all of our tables.

[32:40] I don't think Prisma does this and it is super cool in practice because we are able to use like shared databases and so for projects that, like Total TypeScript has its own database, Epic React has its own database, but something like Astro Party where the partner isn't as active and it's just, it's not even doing commerce, So it doesn't need its own database. We have this idea of a shared database so they all get to share them. And the other really cool thing is if you come into Drizzle config, you can then filter. So you can have a database that has like multiple prefect. This wouldn't work with Prisma.

[33:20] It tries to clobber it, and if you don't have the filter set right, then it will overwrite when you try to push and do schema migrations and that sort of stuff, which is really annoying. Right, so the schema. This is actually importing all of these things and this is what defines the schema. And I wish I could find one. We'll see, I'll try harder.

[33:45] They're all pretty much the same. So if we come in here to the schema, you can see there's some less, right? Like this, I don't think has all of the commerce stuff in it in the video blog. It does have device verification. I was using this as a test bed, but it's like these two, you could literally just import these from the course builder core.

[34:07] And then these schemas would be available inside of your app. And this is another great thing about Drizzle because you can import these. It's not, they didn't invent a new, a whole new text file format to handle this stuff. And I think Prisma's made some progress there where you can have a little more granular schemas, but I don't know, that ship has sailed for me. Cause like, this is nice, right?

[34:29] Like you import these and then inside of here, wherever that is, my SQL table, I forget where the, where const, oh it just exports all of these, so it ends up being, oh, schema, here it is, Oh, not schema, right? So this just, you know, like whatever is in here, whatever we've imported. So if we took all of that away and just had content resource, you know, that would be in your database. And then with a couple of schema setups, it would be very, very compatible, like Zod schemas that we could have shared. It'd be very compatible to have that flow back and forth.

[35:26] And I think that would be pretty cool.