How I Scrape Business Websites to Build Better Directories (Deep Data Enrichment Workflow) Transcript, AI Summary & Key Points

00:00 One of the hardest things about building a directory is getting the right data. I'm not even talking about the basics like the name, address, phone number. That's really easy nowadays. I'm talking about going way deeper and maybe cleaning a CSV or deeply enriching a CSV to get that information that people want to know because without that information, it's really hard to get people to trust your directory and see it as a source of truth and then submit a lead form, which then turns into a lead that you can go and sell

00:22 off. Now, usually the best source of data is going to be the business's website because they have all the information you possibly need. But the problem is that what used to be a very manual process of clicking websites and manually searching for the service or product that I'm building a directory around can now be automated in a scalable way using AI coding tools like Cloud Code and Appify.

00:42 So that's exactly what I'm going to show you today. There's a bunch of reasons why you may want to scrape data off a website over Google Maps. Maybe you're trying to build an events directory or a job board where the data isn't really straightforward to go and scrape compared to a location-based directory. But in this video, I'm going to be scraping data off websites more for the purpose of data cleaning and deep data enrichment for a niche that I've been wanting to build for a long time, which is luxury restroom

01:04 trailers. And I'm not going to lie, the first time I tried to build this, this was a major turnoff. I had to manually look up information on websites to confirm if they offered luxury restroom trailers for rent. And it really just made me put a pause to the project. So, now that I have a solution, I'm just way more willing to go and build this thing out.

01:21 Now, before we start, there are four things that you need prepared if you're going to go and do this. The first one is going to be a Claude Code subscription, which starts at $17 a month. The second one is scraped data. I scraped some sample data off out scraper for this niche. You can use Appify to scrape your data as well, but it is a little bit more expensive compared to Out Scraper.

01:39 So, if you are building a location-based directory, I would just use Out Scraper, especially if it's a nationwide directory with a ton of data that you're working with. The third thing you want is a very clear direction and definition for what you're building your directory around and synonyms to describe that thing. For example, mine's luxury restroom trailers, but people also type in luxury portaotties into Google to find the same exact thing.

02:01 So, you kind of want to find that, but for your own niche. The fourth and last thing that you want going into this is just an understanding of the search intent, right? What drives people's decision-m for the people looking for luxury restroom trailers? They want to know if they can serve their weddings, their outdoor festivals. They also want to know if there's two stall trailers, four stall, eight stall trailers, and that's information that we want to know going into this so that we can tell Claude code to go and

02:24 find that information for us. With that said, let's go ahead and dive into the workflow of scraping websites. So, let's start with the challenges when it comes to building a directory in the luxury restroom trailer niche. The idea is great. I validated it multiple times. It has high search volume, low keyword difficulty. It's also a high ticket niche, right?

02:41 People spend $1,000, $2,000 a day on these rentals. So, it all looks good. and I'm ready to go and build. Starting with the data. So, I went to out scraper like I usually do, use their Google Maps scraper, and this is what we're seeing here. And this is when we start to run into some issues. We have this massive sheet with a bunch of junk data. I see Home Depot, Walmart, and Lowe's.

03:00 There's a bunch of other businesses that don't deserve to be in a luxury restroom trailer directory. Let's just say we manually clean this and remove the obvious ones. we'd still be left with a pretty massive CSV, probably 10,000 listings where we don't really know if they offer luxury restroom trailers, let alone portaotty rentals. And it makes it very timeconuming to go and clean this data cuz now you have to go and manually check on a website, look for any portaotty rentals.

03:22 And then if it does, check if they have luxury restroom trailers. And that doesn't even include the data enrichment side where okay, you found a company that offers luxury restroom trailers. Now you have to go and curate all the information that that one company offers. And so this is exactly the problem that we're trying to solve today. Now for the workflow, I'm not going to clean and enrich every single listing on this sheet.

03:43 That's way too much data and I want to keep this short. I just took a sample of 10 businesses here and I color coded them here so we can kind of see if Claude Code did the task correctly. The green highlighted listings are the ones that do offer luxury restroom trailers verified manually. And this orange one right here does offer portaotty rentals, but they don't have luxury restroom trailers.

04:02 So technically, this should not belong in our directory. Just to show you an example of the final output from a data cleaning perspective. This is what we see. Cloud code went ahead and used Apify to go through every single one of these websites and it created this information right over here. We can see that it now identifies if it is a porta-otty company and then if it is, it'll go and verify if it offers luxury restroom trailers.

04:23 But I didn't stop there. I went a step deeper but now for data enrichment. And I said, "Okay, for all the luxury restroom trailer companies that you found, tell me more." And this is when it found this information right here, which tells us that this business offers two stall trailers, four stall trailers, so on so forth. I think you get the point. So from here, I'm going to fire up Claude Code.

04:42 And I am going to do this quickly. If you want a video where you can kind of do this for the first time and hear out how I learned how to do this, this is the video to watch how I started building with Claude Code in just 14 days. From here, we're going to go ahead and just type in Claude cuz we already have it installed. Now, we're ready to get going.

04:58 And the first prompt is just going to be context. I'm going to tell it a little bit about my task here. I have a CSV of website URLs that I wanted to scrape. So, let me write this and then come back when it's finished. So, this is the prompt that I came up with. You can pause it if you want, but I'm basically telling it about my data cleaning and data enrichment task with Appify.

05:14 And I do talk about my luxury restroom trailer directory idea, the challenges associated with it. And I'm saying that I want to give my CSV full of these website URLs to Claude Code and Appify to basically scrape the website and verify whether or not a business offers portaotty rentals and or luxury restroom trailers. At the very end, I say, "Please start by recommending the best Appify actor that's best for this task."

05:36 And I'm going to go ahead and let this run. And by the way, Appify actor is just another way to say Appify Scraper. We can see that Appify isn't one scraping tool. It's a marketplace for data scrapers. And we can see on the front page, these are the most popular ones. We're probably going to be using this one, website content crawler or something similar to it.

05:54 But there is one for Google Maps. As we can see, it's just a little bit more expensive than Outs Scraper in my experience. So, might as well save a few bucks and just get your data from out scraper and then bring it in this way. So, it came back with a few recommendations for ampify actors, which is interesting because the best choice apparently is the GBT scraper.

06:12 That's not the same ampify actor that I used for my test run, which worked really, really well. And the one that I used was called the Playright Scraper. When you're doing this, just make sure that you explain your task as thoroughly as possible so it understands, you know, the best one to recommend. So, I'm just going to move on and basically tell Claude Code that I used the playright actor before and it worked well, but just start to give me the setup instructions for Ampify and then it will kind of walk us through

06:33 the steps to get this thing working. Sometimes Claude Code might give you the wrong recommendation for actors, but the truth is you learn by just trying it out because the best way to find the best actor for you is to run a small test run because Apify has a free $5 credit and you can try this out to see if you're satisfied with the end output. For the setup, it's pretty straightforward.

06:52 Just go to appify.com, sign up for a free account, get your API token by going to the console, and then just give it to Claude Code. So, we can now connect Claude Code and Appify. Starting with the first one, I'm just going to click get started. And I am going to use a new email. So, I have the $5 credit. Once you create an account, you will see a dashboard like this.

07:10 And then now we can head over to the left-hand side under settings and go to API and integrations. And this is the personal API token that you want. So, I'm just going to go ahead and copy this. I will be rotating this token. I'm just showing you the actual key here just for demonstration purposes. Head back over to Cloud Code. And if you want, you can just give Claude Code your key.

07:29 So, I'm just going to copy and paste it here and say, "Here's my API key." So, I gave it my key and said, "Please set it up." And just confirm that it's good before we move on. Cool. My connection is ready to go. And here it does give us a heads up just for best security practices since I gave my API key. You should rotate it after the project is done.

07:47 So, at this point, it's asking for our CSV with all of the URLs, but I would highly recommend that you give it some more context before you start to run this scrape. And the way to think about it is that Appify right now is the engine that allows us to scrape and crawl all of these website URLs at scale. But Claude Code is the brains of the operation.

08:08 Right now, it doesn't know what your budget is, how many pages that has to go through to find this information. and you can tell it and direct it to look for certain keywords, maybe those synonyms that we prepared, just so it doesn't scrape and look at a 300page website and run up your Apify bill when you really don't need to do that. So, I'm just going to go ahead and write a prompt basically telling it the keywords, the synonyms of the keywords I'm looking for.

08:29 I might even mention some game plan and then I'll get back to you once that prompt is finished. So, this is the prompt that I wrote and you can pause it if you want, but I basically said that I want to do this efficiently so it doesn't cost a ton and we're not scraping pages that are unnecessary. And I basically take time to define what a luxury restroom trailer is because there are some websites that say they have luxury portaotties, but they're not actual trailers.

08:48 So, I just took time to kind of define that and I said there could be one or multiple stalls. Some porta potty companies may call it luxury, but it's not. And then I added to look for some keywords that match the search intent for luxury restroom trailers such as luxury portaotties, luxury bathroom rentals, and anything else you can think of. Lastly, just go ahead and check the homepage and any main menu links on the website so that we're efficient with how many pages we're scraping.

09:12 Awesome. So, it's kind of reiterating its game plan here. Proposed keyword lists looks pretty good to me. It says this should limit most sites to 1 to three pages instead of crawling the entire website. I'm going to tell it can do 1 to seven pages just to get more depth in terms of data, but it's totally up to you. And then I'm just going to give it my CSV.

09:30 So, let me write that prompt and then get back to you. I just said you're welcome to limit to 1 to seven pages for more accuracy. Here's my CSV and use the playright actor and just look at the website URLs. Let's go ahead and see what it can do. The first test run is complete and it did kind of time out here. We can see that four sides timed out. So, I am just going to let it do its thing and finish up these four and then we'll see the results afterwards.

09:50 I think the problem was that I specified it was seven pages as the max. In reality, I think maybe just don't specify. Leave it to Claude code to find that information because I know for a fact one of these M andK portaotties has luxury restroom trailers. I'm just going to keep it vague and say don't worry about the seven-page limit. Just use best practices and be efficient.

10:08 So, we hit a little bit of a snag and despite this working perfectly with the same exact data and the same exact test last night. Um, something kind of went wrong. And so, let me explain what happened. Here are the results from the very first run that we did. And I won't say this is terribly wrong. It just kind of missed this row right here, row 10.

10:26 and there's no data. Either the website timed out continuously or something just went about. It didn't scrape too many pages for most of them. One or two pages, five pages deep for a couple listings here. But other than that, the good news is that it did its job correctly in a sense. It found three out of 10 businesses that offer portaotti rentals. And then under offers luxury trailer, it did catch nestpumping.com that offers portaotty rentals but not luxury restroom trailers.

10:53 And just to kind of show you what I mean, if we go to the website, we do see porta potty rentals right here, but they don't actually offer the trailers that we're looking for. So, at least it caught that, which is awesome. But it just missed this one row. And long story short, I tried to run it again, but I just cleaned up the CSV that I gave Claude Code.

11:09 This is the one that I gave it. It has all of this information, and I just gave it a simplified one. And then this is the second result of what we got. And in these second results, we see that it yet again missed this single row, but it also missed this one right over here. But other than that, the data looks good. So, I guess I'm starting to see some slight inconsistencies when it comes to running Apify.

11:29 And I don't think it's necessarily Appy's fault. I looked it up to see why it would time out. It may be due to the actual website's anti-scraping measures. It could be due to the specific Aify actor that I chose. I would probably test around to see other actors to see if I can get 100% perfect results like I did last night. So, this is kind of a realistic situation of what may happen to you.

11:49 Uh, but just letting you know. But with that said, let's continue on. And now that we have at least some of the correctly mapped out luxury restroom trailer businesses, we can now prompt Claude Code and ask for more information. We want to go deeper and I want to get an idea of their full service offering. So, I'm going to go ahead and just go back to Claude Code.

12:07 Write a prompt and I'll get back to you when it's done. So, I just wrote a quick prompt and I basically said, "I want to hone in on these two restroom trailer businesses that you found." And I want to know the restroom trailer options. Is it two stalls, four stalls? And I want to know this so I can accurately include this data on my directory listings.

12:20 So, go ahead and dig deeper with Appify. And I'm going to let this run. See what it can come up with. Hopefully, this turns out pretty well. All right, that was actually pretty quick, maybe a few minutes. And this is the CSV that we got from an enrichment side. And it looks pretty damn good. So, it has M&K portaotties and A1 rentals. And check out the level of data enrichment that we can use for these listings.

12:41 We have the trailer model, the exact size of the luxury trailer stall count right here, which is super valuable. The configuration, and we even got some pricing information for this second one, A1 rentals. It's $1275 a day, I believe, and 1225 a day for these kind of different offers. So, overall, this is incredibly thorough. So, just to break down my final thoughts on Ampify, I think that there are a few different kind of disadvantages or things that I would consider if you're going to go and use this and one major

13:10 advantage. The major advantage here is that you can actually just have this running overnight. You don't have to really do anything or babysit this. If you wanted to set a budget, for example, of $100 and see where that takes you, then you can go ahead and do that. Uh, we can go to usage right here. And if we look at the bottom, it says custom usage limit.

13:29 You can set this to however much money you want to spend. You do have to upgrade to the monthly paid plan if you want to set this, which kind of sucks. I've done it before, but I think it could be worth it because this $39 a month actually goes towards your credits for your Apify scrapes. So, it's not like this money is wasted. And it also depends on your niche.

13:46 If you have a niche like mine where the data enrichment can go really, really deep, then I think it's absolutely something to consider because this data right here would take a good amount of time to go and get manually. So, for these reasons, it may be worth it for me personally because I want it to be super enriched and I believe that this niche is lucrative enough or has monetization potential enough to make this worth it.

14:09 As far as the cons and some reasons why you may not want to use Aify, I would say the first one you witnessed, which is it can be a little bit glitchy, kind of finicky when it comes to the task accuracy. So, that may happen to you. The second one is the cost. So, what we did today costed $1.35, which isn't much, and it's nice. you get $5 for free, but if you have thousands of listings, then this could get pricey very quickly.

14:31 And the third con is that it's just a little bit slow. Like, this took me a long time, and I think something went wrong with the first run. The second run took a long time, too. It probably took 25, 30 minutes just to run one of those scrapes. So, I've been sitting here waiting for an hour just to see these results. So, if you're looking for something fast and accurate, I'm not sure this is the best option.

14:49 Um, but if you don't mind just letting this run overnight, then this is not that big of a deal. So, that's pretty much it. and at least you have one more tool in your tool belt. So, those are my thoughts on using Appify. Hopefully, that was valuable. I think this does unlock so many potential directory ideas that I haven't really explored. I've kind of only built location-based directories, but I've always been curious about these AI tool directories where I saw a board game directory last year that was absolutely

15:13 crushing it, doing like multiple millions a year. So, this now allows me to go and get data for those types of directories. And maybe you're building one yourself. So, if you are, let me know in the comments below. And also feel free to let me know what other challenges you have when it comes to manipulating or acquiring the right data. I can always make more videos around this topic because I have a feeling a lot of people stop when it comes to the data side of directories.

15:36 But that's it for now and as always I appreciate you watching. Thanks so much for the support and I'll catch you in another video. See you.