SEO Site Monitoring

Audits Hero Image
When fruit goes bad it rots, telling you something is wrong and needs throwing. Your website is being worked on by lots of different people, monitoring allows you to similarly know when something is broken.

What is Site Monitoring and what is involved?

Modern, enterprise websites often release changes to their code several times a day, preferring to tolerate mistakes but not taking a long time to find them. My site monitoring platform allows for sites with millions of URLs to be monitored daily and includes a helpful Looker report for stakeholders and all the requisite data to replicate issues.
Crawling

Crawling

Crawling at an hourly, daily or weekly cadence is setup and run so to find and aggregate issues on the site.
Reporting

Reporting

Data from the crawl is aggregated and feed into a Looker report, which showcases both current versus previous crawl numbers and trends for 30 days previous. Additionally all of the data required to fix the indicated issues is included and available in a onedrive folder which updates after every crawl.
Data Gathering

Data Gathering

A secondary but useful side effect of crawling your website so often is I can also pull and aggregate data from across your site. Want a count of pages which have ‘Staff Writer’ instead of an actual author, this can be collected and provided daily.

How the Process Works

Step 1

Setup

Working with your engineering teams I create a route for daily crawling, which typically involves whitelisting my IP, UserAgent and header token. Once whitelisted I setup the crawls and begin a series of tests to confirm we won’t be blocked.
Step 2

Reporting and training

I connect the aggregated data collected from the crawls to a looker template which outlines current vs previous crawl deltas and provides a 30 day lookback of crawl stats. In recognition of the fact people ignore reporting they do not understand, I am happy to provide training.
Step 3

Alerting

Once reporting is being populated we begin looking at alerting. I will work with you to setup thresholds and/or categories that must always result in alerts, pushing them either by email or slack messages.

FAQ's

Do I need your help if I have a Product/Project manager?

I read this as a question of whether a generalist can do the job of a specialist.

Whilst a lot of things in SEO are arguably just being neat and in-keeping with commonplace best practices, it remains that considering how obvious and common these consideratiozs are they do not really create any sort of advantage in the market.

Anticipating what are going to be the most impactful of the more mature tactics is where an expert comes into play, allowing you to leverage their skills, experiences and training to avoid waste and get straight to the stuff that works.

Can I not focus on SEO after the site is live?

Technically speaking of course you can. The issue with this train of thought is the waste.

By not building it right the first time you waste time on the first iteration, potentially re-building or throwing away code and resources.

By sending live a site which is not optimised you waste creating a positive impression with search engines whilst the site is new. Depending on the issue, this can take quite a long time to unpick/solve (Google [Discovered – Currently Not Indexed]).

Lastly, by making your engineers work on code which they then throw away it wastes their enthusiasm for the project. Nobody wants to throw away something they spent a lot of time working on, least of all engineers that are motivated to do good work.

Should I have everything off the root or in directories?

This is admittedly a bit of a tangent but it comes up a lot! The short answer is both can work but it’s typically easier to work with your site when organised into directories. Spending 3 days tagging pages is not fun, and as you add more you need the team to remember to do it. Conversely, putting a page about puppies in the /dogs/ folder is obvious and doesn’t require any additional work.

A pro-tip here is also that when pulling data from search console you can get more from their API if you have profiles setup for each directory on your website – this is not an option for flat sites.

How to Contact Me
Drop me an email
Visit my Contact Page