If you want to take advantage of one of Screaming Frog’s latest and most helpful features, or if you are just trying to get your content to rank better in search, this post is here to help. Using AI to analyze site architecture is no longer a futuristic concept, it is a practical workflow you can implement today.
In this post, I will cover how to use Screaming Frog’s:
- API connector for LLMs
- Built-in LLM prompt for creating embeddings
- Content Clustering tool
I will then show you how you can use Google Colab to leverage that data and get recommendations for internal linking that can help improve the topical relevance and topical authority of your site and its content. I have also included the necessary code to execute this analysis at no cost.
Configuring the API Connector for LLMs
The foundation of this workflow is the API connector, which bridges Screaming Frog with Large Language Models. To set this up, navigate to Configuration, then API Access, and select AI.
In the AI configuration window, you will see a list of providers including OpenAI, Google Gemini, Anthropic, and Ollama. For most users, OpenAI or Gemini are the most reliable choices for generating embeddings. Once you select your provider, you must provide a valid API key.
After entering your key in the Account Information tab, you must click the Connect button. This is a critical step; Screaming Frog will verify the connection and display a success message. If you skip this, the AI prompts will not execute during the crawl. Ensure you have sufficient credits in your API provider account, as generating embeddings for thousands of pages incurs a small cost.
Setting Up the Embedding Prompt
Embeddings are mathematical representations of your content that allow computers to understand the relationship between different topics. Screaming Frog has simplified this by providing a library of pre-configured prompts.
Navigate to Configuration, then Prompt Configuration. Click the Add from Library button and select the option for extracting embeddings from page content. This prompt instructs the LLM to look at your page text and return a vector string.
Before running your crawl, you must ensure that the software is capturing the data the LLM needs to see. Go to Configuration, followed by Spider, and then Extraction. You must enable Store HTML and Store Rendered HTML. This ensures the full text of your pages is available for the embedding process.
Generating Content Clusters
Once your crawl is finished and the AI tab shows that all embeddings have been processed, you can visualize your site’s topical structure.
Go to Configuration, then Content, and select Embeddings. Ensure the enable embeddings functionality box is checked and that your semantic similarity threshold is set. A common starting point is 0.8 or higher for identifying closely related content.
After the crawl completes, you must run a Crawl Analysis for the clusters to populate. Once finished, navigate to the Visualizations menu and select Content Cluster Diagram. This provides a 2D map of your website where each dot represents a page. Pages that are close together are topically related, while outliers represent content that may be disconnected from your primary pillars.
Leveraging Google Colab for Internal Linking Recommendations
The visualization is helpful for a bird’s eye view, but for actionable internal linking, we need to process the raw data. By exporting your embeddings, you can use a Python script to find every page’s most relevant topical neighbors that are not yet linked.
Follow these steps to prepare your data:
- Export the Content tab as a CSV file.
- Ensure your CSV has at least two columns: Address and the column containing your embeddings data.
- Open a new notebook in Google Colab.
Python Code for Semantic Mapping
Paste the following code into a cell in your Google Colab notebook to calculate the similarity between all pages on your site and identify linking opportunities.
import pandas as pd
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
from google.colab import files
# Configuration: Adjust these variables as needed
TOP_K = 10 # Number of recommendations per URL
MIN_SIMILARITY = 0.70 # Minimum cosine similarity threshold (0.0 to 1.0)
# Upload your Screaming Frog export
print("Please upload your Screaming Frog 'Custom JavaScript' or 'Content' export.")
uploaded = files.upload()
filename = list(uploaded.keys())[0]
# Load data
df = pd.read_csv(filename)
# Clean data: Ensure you point to the correct column name for embeddings
# Screaming Frog usually names this based on the prompt used
embedding_col = 'Extract embeddings from page content'
# Convert string embeddings to numpy arrays
def parse_embedding(x):
if pd.isna(x) or str(x).strip() == "":
return None
try:
# Handles various string formats like [0.1, 0.2] or just 0.1, 0.2
clean_str = str(x).replace('[', '').replace(']', '').strip()
return np.fromstring(clean_str, sep=',')
except:
return None
df['vector'] = df[embedding_col].apply(parse_embedding)
df = df.dropna(subset=['vector'])
# Ensure vectors are not empty and have consistent dimensions
matrix = np.vstack(df['vector'].values)
# Calculate Cosine Similarity
print(f"Calculating similarity matrix for {len(df)} URLs...")
sim_matrix = cosine_similarity(matrix)
# Find Top-K recommendations per URL
results = []
for idx, row in df.iterrows():
# Get similarity scores for this specific row
scores = list(enumerate(sim_matrix[idx]))
# Sort by score (descending)
# Filter out the page itself (where target_idx == idx)
sorted_scores = sorted(scores, key=lambda x: x[1], reverse=True)
filtered_scores = [s for s in sorted_scores if s[0] != idx]
# Take the top K results
top_k_scores = filtered_scores[:TOP_K]
for target_idx, score in top_k_scores:
if score >= MIN_SIMILARITY:
results.append({
'Source URL': row['Address'],
'Suggested Target': df.iloc[target_idx]['Address'],
'Similarity Score': round(float(score), 4)
})
# Export findings
output_df = pd.DataFrame(results)
output_filename = 'internal_link_recommendations.csv'
output_df.to_csv(output_filename, index=False)
print(f"Analysis complete. Found {len(output_df)} total recommendations.")
files.download(output_filename)
Implementing the Recommendations
The resulting CSV will give you a list of source pages and suggested targets based on their mathematical similarity. When reviewing these, look for pages in the same cluster that do not currently link to one another.
By adding these links, you reinforce the topical signals you are sending to search engines. This method ensures that your internal linking strategy is based on actual content depth and semantic relevance rather than simple keyword matching.
Validating the Implementation
Once the recommended links are live on your site, you must verify that they have been implemented correctly and are visible to search engine crawlers.
The most effective way to do this is to perform a follow-up crawl with Screaming Frog. After the crawl, navigate to the Internal tab and sort by the Inlinks column for your target pages. You should see an increase in the number of unique inlinks. Additionally, check the Crawl Depth of your key pages. A successful implementation often reduces the number of clicks required to reach important content from the homepage.
You should also use the Crawl Comparison feature in Screaming Frog to compare your baseline crawl with the post-implementation crawl. This will highlight exactly which URLs received new links and ensure no unintentional errors, such as broken links or redirect chains, were introduced during the process.
Summary and Next Steps
This workflow transforms internal linking from a manual, subjective task into a data-driven strategy. By combining Screaming Frog’s AI connector with Python analysis, you can ensure your site architecture mirrors the topical relationships that modern search engines look for.
To maintain your topical authority, consider the following next steps:
- Schedule a quarterly semantic crawl to identify new linking opportunities as you publish more content.
- Monitor Google Search Console for changes in impressions and rankings for the pages you have newly interlinked.
- Audit your anchor text distribution to ensure your new links are using descriptive, relevant keywords that provide context to both users and bots.
By consistently applying these semantic insights, you will build a more resilient and authoritative website.