Source code and an installer can be found on GitHub at:

• https://github.com/nazuke/SEOMacroscope/releases/tag/v1.7.6.1

Please check the downloads page for more recent versions.

New features in this release include:

• Crawl configuration presets.

Bug fixes

• Nothing significant for this release.

Please report issues at https://github.com/nazuke/SEOMacroscope/issues.

As we all know, the keywords meta tag hasn’t been used by the major search engines for quite some time now.

However, in a recent feature request, it does appear that this particular tag still lingers in some CMS systems. In particular, the request in question uses this feature as a part of the editorial process to ensure that the keywords that are supposed to be in the page’s body text, are actually used in the page’s body text.

Naturally, as these pages are edited by users, and as it’s up to the users to ensure that their edits are consistent, this doesn’t always work as intended.

To that end, there’s a minor feature in SEO Macroscope now that analyzes the contents of the keywords meta tag, and verifies whether or not the keywords are mentioned in the title tag, the description tag, and the body text itself.

This can be used to determine whether or not the published web pages are consistent or not.

The process itself is automatic.

First of all, carry out teh crawl process in the normal way. Speed things up by only crawling HTML pages. Once that’s done, the crawled document set will be available and the Keywords Presence tab can be selected.

The Keywords Presence tab show one row for each keyword-related problem on each page.

#h2 Empty Keywords Meta Tag

If the keywords meta tag is entirely empty, then this will immediately be flagged up as a problem.

For most sites, this isn’t going to be a problem. In this case however, it means that the page author has forgotten to specify which keywords they’re trying to target with the page in question.

#h2 Malformed Keywords Meta Tag

Another problem that arises may be where the author has pasted the wrong data into the keywords meta tag, or simply entered text that cannot be properly parsed.

The keywords are defined as a comma-separated values list. This means that of the keyword list cannot be properly parsed into a list of terms, then it will be flagged as an error.

#h2 Keyword Missing in Body Text

If the keywords meta tag itself is fine, then the keywords themselves are analyzed, and then cross-referenced against the text in the page body, the title tag, and the description tag.

If a keyword is specified in the keywords meta tag, but cannot be found in the body text, then it is flagged as such:

Likewise, if the keyword is present, then it is also reported, for clarity’s sake:

#h2 Keyword Missing in Title and Description

Similarly, if the keyword is missing from the title tag, or is present, then it is reported as such:

Likewise, if it is missing from the description tag, or present, then it is reported too:

With the title and description tags, it is up to the disgression of the author as to whether or not this is a problem.

Source code and an installer can be found on GitHub at:

• https://github.com/nazuke/SEOMacroscope/releases/tag/v1.7.6.0

Please check the downloads page for more recent versions.

New features in this release include:

• Titles and descriptions are now included in the keywords meta tag analysis. Additionally, some malformed keywords meta tags will be reported as such.
• Some refactoring has been done behind the scenes to try and make the crawled document collection a little more efficient.
• New Recent URLs sub-menu under in the File menu.
• New Status Code columns in the Links and and Hyperlinks overview lists.
• Anchor Text columns added to broken links reports.
• Experimental Save Session feature, to save and reload the current crawled session.

Bug fixes

• Fixed error handling in AnalyzeKeywordPresence.
• Added preferences option to re-fetch linked documents from external sites that initially return a 404 on a HEAD request. This helps to verify links to external sites that have mis-configured webservers, at the expense of network bandwidth.

Please report issues at https://github.com/nazuke/SEOMacroscope/issues.

Source code and an installer can be found on GitHub at:

• https://github.com/nazuke/SEOMacroscope/releases/tag/v1.7.5.0

Please check the downloads page for more recent versions.

New features in this release include:

• A recent request was made for processing the contents of the legacy “keywords” meta tag. This feature is enabled by default, with the results available in the new Keywords Presence display list. Very briefly, the contents of the keywords meta tag is examined, and then the presence or absence of each keyword in the page body text is reported. Currently, this only applies to the body text; other elements, such as the title tag, are not processed.
  • Normally, I would advise against using the keywords meta tag in any new websites; however it appears that this meta tag is still used by some CMS platforms, and is a reasonable method to check that keywords that should be present in the body text are actually there, or not.
  • Please note that this analysis step is separate to the existing keywords analysis; that analysis ignores the keyword meta tag entirely, and operates purely on the body text alone.

Bug fixes

• Licence window was broken.
• Preferences window resized for smaller screens.

Please report issues at https://github.com/nazuke/SEOMacroscope/issues.

Source code and an installer can be found on GitHub at:

• https://github.com/nazuke/SEOMacroscope/releases/tag/v1.7.4.0

Please check the downloads page for more recent versions.

New features in this release include:

• There is a new option to probe parent directories for each URL found on an internal site. This builds a new set of URLs to crawl, by taking the current URL, and progressively stripping off each rightmost element until it reaches the root. Each stripped URL is then added to the list of URLs to crawl.
• The body text word counter has been improved, and unit tests written.
• Regular expression data extraction now works on PDF documents.
• PDF embedded link extraction and following has been improved.

Bug fixes

• Not a bug as such, but the check for update phone home function now more precisely checks the current and updated version numbers, instead of doing a simple equals comparison.
• Keyword analysis is now skipped when humans.txt, and some other page types 404.
• Absolute URL handling in robots.txt has been improved.

Please report issues at https://github.com/nazuke/SEOMacroscope/issues.

So when I started work on SEO Macroscope, I recycled this idea to test out the extraction functionality, Excel and CSV reporting, and so on.

In usage, it’s very simple. When scanning a website, SEO Macroscope will automatically look for email and telephone links on each HTML page, and extract them out into a list associated with each page.

When the scan is complete, the extracted email addresses and telephone numbers can be found under their respective tab views, and the pages on which they appear.

In addition, the email addresses and telephone numbers can be exported as Excel or CSV reports with the Contact Details Report.

The extraction process described above only operates on link elements; if you require more sophisticated extraction of email addresses or telephone numbers, then the Data Extractors functions can be used instead.

For example, data extractors can be configured using regular expressions to extract telephone number from the page body text, even if they do not appear in link elements. Likewise, the same is true for email addresses.

Source code and an installer can be found on GitHub at:

• https://github.com/nazuke/SEOMacroscope/releases/tag/v1.7.3.0

Please check the downloads page for more recent versions.

New features in this release include:

• Where possible, Author fields are extracted from HTML and PDF documents.
• The Page Metadata Excel report has a new worksheet that combines the crawled author, title, description, and keywords fields.
  • This can be useful when crawling a list of PDF documents, as it extracts that information into a single worksheet.
• A simple check for update feature has been added. This will show an alert if a new version of SEO Macroscope appears to be available.
• HTML page character set sniffing has been enhanced.

Bug fixes

• Rewrote the redirect chain analysis code, so that the redirect chain analysis should now be more complete for each crawl. Previously, the redirect chain list was built from the crawled document collection, which meant that some redirects were missing if they had not been crawled yet. Not, an explicit HEAD request is executed for each document that redirects, until no more redirects are encountered.
• There was a locking fault in the crawled document collection, that caused some documents to never be fetched.

Please report issues at https://github.com/nazuke/SEOMacroscope/issues.

Source code and an installer can be found on GitHub at:

• https://github.com/nazuke/SEOMacroscope/releases/tag/v1.7.2.0

Please check the downloads page for more recent versions.

Bug fixes

• Incorrect behaviour of progress dialogues.
• Removed memory guard on sitemap generators, that may have prevented sitemap generation under certain circumstances.

Please report issues at https://github.com/nazuke/SEOMacroscope/issues.

Source code and an installer can be found on GitHub at:

• https://github.com/nazuke/SEOMacroscope/releases/tag/v1.7.1.0

Please check the downloads page for more recent versions.

New features in this release include:

• There is a new hyperlink ratio feature found in the document details panel, and in the overview Excel and CSV reports. This calculates the percentage value for the number of hyperlinks in and out of a particular document, within the crawled collection. It does not include links from third-party sites not in the crawled collection.
• The web proxy settings may now now the systems configured proxies.

Bug fixes

• A malformed User-Agent HTTP Header caused some websites to not be crawled at all.

Please report issues at https://github.com/nazuke/SEOMacroscope/issues.

Simple ETag and Checksum Logging

The simplest of these is achieved by recording the ETag HTTP Header, if it’s returned by the remote web server, and identify which of the crawled URLs appear to have the same ETag value.

Similarly, for any documents that SEO Macroscope actually downloads (such as HTML, PDFs, etc…), a checksum value is computed. Different URLs that have the same checksum are a strong indicator that the content is exactly the same.

Finding duplicates by Levenshtein Edit Distance

Another technique that I’ve used is Levenshtein Edit Distance measuring; leveraging Dan Harltey’s Fastenshtein implementation. Currently, SEO Macroscope will only apply this to documents that have body text in them, such as HTML pages and PDFs.

Briefly, what that means is that if you have multiple pages on your site that are very closely similar, but perhaps with a few minor differences, then these may be detected and reported upon. For example, if two pages are very closely similar, but perhaps they were rendered with very slightly different text in them somewhere, then they will not have matching checksums, but they may be similar enough to fall within the Levenshtein Edit Distance threshold that you specify.

A typical example may be an ecommerce site, that presents much the same content under different URL variations.

The only drawback is that this is quite an intensive process if there are a lot of pages on your site; so it may be necessary to restrict spidering to a subset.

The SEO Macroscope preferences includes options to specify the initial similarity of the documents to apply the Levenshtein algorithm to. If documents that fall within the parameters are found, they will be reported.

Export Excel Reports

In all cases, the ETags, checksums, and Levenshtein Edit Distance values can be found by exporting a Duplicate Content Report from the Reports menu, after completing a crawl of your website.

Please note that if you have enabled Levenshtein Edit Distance in the preferences, it may take quite some time for the report to be generated.