Mwmbl is continuing to develop its search engine with new features and ways for the community to get involved. One of these features is Seed Search, which helps improve search results while expanding Mwmbl’s own search index.

Seed Search helps Mwmbl find relevant web pages using its existing search index and external search providers. It can help users find pages that Mwmbl’s own index might not have found yet.

Mwmbl is an independent, open-source, non-profit search engine. Its index is smaller than those of major commercial search engines, so it can sometimes miss pages that users are looking for. Seed Search is intended to help address this problem while supporting the growth of Mwmbl’s index.

How does Seed Search work?

Seed Search works through several steps:

  1. Search: Mwmbl searches its own index and uses external search results to find additional pages.
  2. Collect results: Results from different sources are gathered together.
  3. Rank results: The results are scored and ordered to help users find relevant pages.
  4. Discover pages: Pages found through searches can help Mwmbl discover content that is not yet in its index.
  5. Grow the index: Newly discovered pages can help expand Mwmbl’s index and improve future searches.

This means Seed Search can help users find pages while also helping Mwmbl improve its search index over time.

What is seed crawling?

Seed crawling helps Mwmbl discover and index new web pages. A crawler starts with selected URLs, retrieves pages it is allowed to access, and extracts useful information that can be processed for the search index.

The process can be described in several steps:

  1. Choose seed URLs: The crawler starts with URLs supplied as starting points.
  2. Fetch pages: It retrieves web pages that it is permitted to crawl.
  3. Extract information: It collects useful page content and links to other pages.
  4. Discover more pages: Links found on crawled pages can help identify additional URLs to visit.
  5. Expand the index: Information from discovered pages can be processed and added to Mwmbl’s search index.

Seed crawling and Seed Search are related, but they serve different purposes. Seed Search helps find and rank results for a query, while crawling helps discover and collect pages that can become part of the search index.

Why is seed crawling useful?

A search engine needs an index of web pages to return useful results. Seed crawling helps Mwmbl discover pages it might not already know about and expand the collection of content available to search.

As the index grows, Mwmbl can potentially offer a wider range of results. Crawling can also help discover links to additional pages, allowing the index to grow beyond its original starting URLs.

Membership and quotas

Mwmbl has membership tiers that offer different levels of access to Seed Search and other features.

The membership allowances described in the project’s tier proposal include:

  • Sprout: 300 Seed Search queries per month.
  • Sapling: 1,500 Seed Search queries per month.
  • Canopy: Includes Sapling benefits, plus up to one million pages crawled per month against the member’s username.

These tiers provide different levels of access while helping manage the resources needed to operate the service. The crawling allowance in Canopy also gives members an additional way to contribute to the growth of Mwmbl’s index.

Membership benefits and quotas may change as Mwmbl continues development. Check the current membership details before publishing these figures as confirmed limits.

Privacy

Mwmbl is focused on building independent search technology. When Seed Search uses external providers, query text may be sent to those providers to retrieve or rank results.

Users should check Mwmbl’s current documentation to understand how queries are processed and what information is shared with external services.

What’s next?

Seed Search and seed crawling are part of Mwmbl’s ongoing work to improve search quality and expand its independent index.

By combining search capabilities with ways to discover new pages, Mwmbl can help users find content while continuing to grow its searchable collection. As the project develops, its functionality, crawling capabilities, quotas, and membership benefits may change.

Finally…

Mwmbl continues to explore ways to make independent search more useful while keeping its technology open source. Seed Search and seed crawling are examples of how the project can help users find pages today while contributing to a larger index for future searches.

For more information, visit the Mwmbl website or join the Mwmbl Matrix room.