SEO Site Monitoring
When fruit goes bad it rots, telling you something is wrong and needs throwing. Your website is being worked on by lots of different people, monitoring allows you to similarly know when something is broken.
What is Site Monitoring and what is involved?
Modern, enterprise websites often release changes to their code several times a day, preferring to tolerate mistakes but not taking a long time to find them. My site monitoring platform allows for sites with millions of URLs to be monitored daily and includes a helpful Looker report for stakeholders and all the requisite data to replicate issues.
Crawling
Crawling at an hourly, daily or weekly cadence is setup and run so to find and aggregate issues on the site.
Reporting
Data from the crawl is aggregated and feed into a Looker report, which showcases both current versus previous crawl numbers and trends for 30 days previous. Additionally all of the data required to fix the indicated issues is included and available in a onedrive folder which updates after every crawl.
Data Gathering
A secondary but useful side effect of crawling your website so often is I can also pull and aggregate data from across your site. Want a count of pages which have ‘Staff Writer’ instead of an actual author, this can be collected and provided daily.
How the Process Works
Step 1
Setup
Working with your engineering teams I create a route for daily crawling, which typically involves whitelisting my IP, UserAgent and header token. Once whitelisted I setup the crawls and begin a series of tests to confirm we won’t be blocked.
Step 2
Reporting and training
I connect the aggregated data collected from the crawls to a looker template which outlines current vs previous crawl deltas and provides a 30 day lookback of crawl stats. In recognition of the fact people ignore reporting they do not understand, I am happy to provide training.
Step 3
Alerting
Once reporting is being populated we begin looking at alerting. I will work with you to setup thresholds and/or categories that must always result in alerts, pushing them either by email or slack messages.
FAQ's
What will I know after this report that I don't know now?
What follows this analysis?