Step 1: Set up the Python Environment#

Ensure Python 3 is installed. Install necessary libraries. For parsing HTML, BeautifulSoup is a common choice. The standard json library handles JSON output.

1
pip install beautifulsoup4

Step 2: Write the Content Scraper/Parser#

The core of the script involves iterating through content files and extracting data.

1
import os
2
import json
3
from bs4 import BeautifulSoup
4

5
def extract_content(filepath):
6
    """Reads a file and extracts content."""
7
    # Placeholder: Implement logic based on file type (HTML, Markdown source, etc.)
8
    # For HTML:
9
    try:
10
        with open(filepath, 'r', encoding='utf-8') as f:
11
            soup = BeautifulSoup(f, 'html.parser')
12

13
        title_tag = soup.find('title')
14
        title = title_tag.string if title_tag else os.path.basename(filepath)
15

16
        # Example: Extract text from a specific article container
17
        article_body = soup.find('article') # Adjust selector based on site structure
18
        body_text = article_body.get_text(separator=' ', strip=True) if article_body else ""
19

20
        # Determine the URL (requires mapping file paths to site URLs)
21
        # This mapping logic is site-specific. Example: remove file extension, replace directory separator
22
        base_url = "https://your-site.com/" # Replace with actual base URL
23
        relative_path = os.path.relpath(filepath, 'path/to/your/content/directory') # Adjust path
24
        url = base_url + os.path.splitext(relative_path)[0] + '/' # Adjust for permalink structure
25

26
        return {
27
            'id': url, # Use URL as a unique ID
28
            'title': title,
29
            'body': body_text,
30
            'url': url
31
        }
32
    except Exception as e:
33
        print(f"Error processing {filepath}: {e}")
34
        return None
35

36
def collect_files(directory, extensions=['.html']):
37
    """Collects files with specified extensions from a directory."""
38
    files_list = []
39
    for root, _, files in os.walk(directory):
40
        for file in files:
41
            if os.path.splitext(file)[1].lower() in extensions:
42
                files_list.append(os.path.join(root, file))
43
    return files_list
44

45
# Example Usage:
46
# content_directory = 'path/to/your/static/site/output' # Directory containing HTML files
47
# html_files = collect_files(content_directory)
48
# documents = [extract_content(f) for f in html_files if extract_content(f)] # Filter out errors

• Insight: Parsing HTML requires careful consideration of the site’s structure to ensure only relevant content is extracted. Using specific CSS selectors or traversing the DOM based on known structural elements (like an <article> tag) is more reliable than extracting all text.
• Data Point: A typical site might have 80% of its HTML dedicated to navigation, headers, footers, and sidebars. Efficient parsing reduces the index size and improves search relevance by focusing on core content.

Step 3: Generate the Lunr.js Index Data Structure#

The documents list generated in Step 2 is already in the required format: a list of dictionaries, each representing a document with fields like id, title, body, and url.

1
# 'documents' is the list of dictionaries from Step 2
2
# Example structure:
3
# documents = [
4
#     {'id': 'https://site.com/post1/', 'title': 'Post One', 'body': 'Content of post one...', 'url': 'https://site.com/post1/'},
5
#     {'id': 'https://site.com/page2/', 'title': 'Page Two', 'body': 'Content of page two...', 'url': 'https://site.com/page2/'},
6
#     # ... more documents
7
# ]

Step 4: Save the Index as JSON#

Save the documents list to a JSON file.

1
# Example Usage (continuing from Step 2 & 3):
2
# output_index_file = 'static/search_index.json' # Where the JSON file will be saved in your static site
3

4
# with open(output_index_file, 'w', encoding='utf-8') as f:
5
#     json.dump(documents, f, ensure_ascii=False, indent=2)
6

7
# print(f"Index generated successfully to {output_index_file}")

• Tip: Using ensure_ascii=False ensures non-ASCII characters (like accented letters) are saved correctly, and indent=2 makes the JSON file human-readable for debugging (though increasing file size). For production, remove indent.

This Python script is a build tool. It runs before deployment to create the search_index.json file, which is then deployed alongside the static site files.

Step-by-Step Guide: Integrating Lunr.js into the Static Site

This involves adding JavaScript to the static site’s front-end code.

Step 1: Include Lunr.js Library#

Download Lunr.js or link to a CDN. Include it in the site’s HTML, typically at the end of the <body> tag or in the <head>.

1
<script src="https://unpkg.com/lunr/lunr.js"></script>

Step 2: Load the Generated JSON Index File#

Fetch the search_index.json file generated by the Python script. This is an asynchronous operation.

1
let searchIndex;
2
let documents = []; // Store the full document data
3

4
fetch('/search_index.json') // Path to your JSON file
5
    .then(response => response.json())
6
    .then(data => {
7
        documents = data; // Store the full document data
8

9
        // Create the Lunr index
10
        searchIndex = lunr(function () {
11
            this.ref('id'); // 'id' field from the JSON is the unique identifier
12
            this.field('title', { boost: 10 }); // Boost title matches
13
            this.field('body');
14
            this.field('url'); // Can index URL too if needed for search terms
15

16
            documents.forEach(function (doc) {
17
                this.add(doc);
18
            }, this);
19
        });
20
    })
21
    .catch(error => {
22
        console.error('Error loading search index:', error);
23
    });

• Insight: Loading the index asynchronously prevents blocking the main thread and keeps the page responsive while the file is downloaded and processed.
• Data Point: The size of the search_index.json file directly impacts the time required to download and process it. For sites with hundreds or thousands of documents, managing index size becomes important. Techniques like indexing only essential fields or creating multiple smaller indices might be considered for very large sites.

Step 3: Implement the Search UI and Logic#

Add an input field for search queries and an area to display results. Add JavaScript to handle user input and perform the search using Lunr.js.

1
<input type="text" id="search-input" placeholder="Search site...">
2
<ul id="search-results"></ul>

1
const searchInput = document.getElementById('search-input');
2
const searchResults = document.getElementById('search-results');
3

4
searchInput.addEventListener('input', function(event) {
5
    const query = event.target.value;
6

7
    // Clear previous results
8
    searchResults.innerHTML = '';
9

10
    if (query.length < 2) { // Require at least 2 characters to search
11
        return;
12
    }
13

14
    if (searchIndex) {
15
        try {
16
            const results = searchIndex.search(query); // Perform the search
17

18
            if (results.length > 0) {
19
                // Display results
20
                results.forEach(result => {
21
                    // Find the full document data using the result ref (id)
22
                    const doc = documents.find(d => d.id === result.ref);
23
                    if (doc) {
24
                        const li = document.createElement('li');
25
                        li.innerHTML = `<a href="${doc.url}">${doc.title}</a>`; // Example output
26
                        searchResults.appendChild(li);
27
                    }
28
                });
29
            } else {
30
                const li = document.createElement('li');
31
                li.textContent = "No results found.";
32
                searchResults.appendChild(li);
33
            }
34
        } catch (e) {
35
             console.error("Search error:", e);
36
             // Handle potential errors during search parsing (e.g., invalid query syntax)
37
             const li = document.createElement('li');
38
             li.textContent = "Error performing search. Please try a different query.";
39
             searchResults.appendChild(li);
40
        }
41
    } else {
42
        // Index not yet loaded
43
        const li = document.createElement('li');
44
        li.textContent = "Search index not yet loaded. Please wait.";
45
        searchResults.appendChild(li);
46
    }
47
});

• Tip: Add debounce or throttle to the input event listener for performance on large indices, preventing searches on every single keystroke.
• Usability: Providing feedback while the index loads or when no results are found enhances the user experience.

Refining the Search Experience

Several techniques can improve the quality and performance of the client-side search:

• Field Weighting: As shown in the Lunr.js initialization, weighting fields (this.field('title', { boost: 10 })) makes matches in the title field more relevant than matches in the body. This prioritizes results where the search term is a primary topic.
• Stop Words and Stemming: Lunr.js includes built-in support for removing common words (stop words like “the”, “a”) and reducing words to their root form (stemming, e.g., “running”, “ran” become “run”). These features are typically enabled by default and improve search accuracy by matching variations of a word.
• Handling Large Indices: For very large static sites (thousands of pages), a single large JSON index can be slow to download and process. Potential strategies include:
  • Creating multiple smaller indices (e.g., per section or year) and searching only the most relevant ones first.
  • Only indexing titles and maybe the first paragraph of content, keeping the index size down.
• UI/UX Improvements: Implementing features like search suggestions, highlighting search terms in results, and a dedicated search results page rather than just an inline list can significantly improve usability.

Case Study Example

A technical documentation website for an open-source project, built using Sphinx (an SSG), required offline search capabilities for users downloading documentation archives. Implementing a server-side search was deemed too complex and added an unnecessary dependency.

The solution involved:

1. A Python script that ran after the Sphinx build process. It parsed the generated HTML files, extracting the page title, URL, and the main content from specific HTML divs unique to the documentation pages.
2. The script formatted this data into a JSON list.
3. The Lunr.js library was included in the Sphinx HTML theme.
4. JavaScript code fetched the generated JSON index, initialized Lunr.js, and powered a search box in the site’s header, displaying results in a drop-down list.

This approach successfully delivered fast, client-side search within the downloaded documentation archives, fulfilling the offline requirement without any server infrastructure, aligning perfectly with the project’s static documentation build process. The index size remained manageable for several hundred documentation pages.

Key Takeaways

• Building static site search with Python and Lunr.js leverages Python for offline index generation and Lunr.js for client-side search execution.
• This approach offers performance, cost, and simplicity benefits compared to server-side search solutions for static websites.
• The process involves a Python script parsing static content (like HTML), extracting key data (title, body, URL), and formatting it into a JSON index file.
• Client-side integration requires including the Lunr.js library and JavaScript code to fetch the JSON index, initialize Lunr.js, and handle search queries and result display.
• Effective content parsing in Python, considering HTML structure, is critical for accurate index generation.
• Optimizing index size and using Lunr.js features like field weighting and stemming improve search relevance and performance.
• Asynchronous index loading in JavaScript is recommended for better user experience.
• This method is suitable for static sites where maintaining a simple architecture and minimizing server dependencies are priorities.