Only few search engines index the Web at scale. Third parties who want to develop downstream applications based on web search fully depend on the terms and conditions of the few vendors. The public availability of the large-scale Common Crawl does not alleviate the situation, as it is often cheaper to crawl and index only a smaller collection focused on a downstream application scenario than to build and maintain an index for a general collection the size of the Common Crawl. Our goal is to improve this situation by developing the Open Web Index.
The Open Web Index is a publicly funded basic infrastructure from which downstream applications will be able to select and compile custom indexes in a simple and transparent way. Our goal is to establish the Open Web Index along with associated data products as a new open web information intermediary.
https://downloads.webis.de/publications/papers/hendriksen_2024.pdf
This paper seems to give a good, quick overview.
It looks to be the usual EU tech project. Doing more to achieve less in a desperate, hopeless attempt to make up for the stupidity and greed of European elites.