Has anyone thought of making a Web Crawler/Search / Scraping VM?

General Discussion
1

I thought Qubes could be an ideal platform to make a disposable web-scraping, crawler or search VM.

Has anyone else thought of this or do this or have any tips / thoughts?

I am in the planning stages and am looking at

Scraping:

Selenium / Tor Browser Selenium

The idea would be to set it to scrape a site in a disposable then transfer the data to a vault.

each crawl in fresh disposable VM when the job finishes, the VM is destroyed, taking all session data, cookies, cached scripts, and potential malware with it. If the target site exploits a browser vulnerability, the attacker gains nothing persistent.

and for a web crawler:

and for a search engine:

2

That would be cool. Share your code if you actually end up writing a search engine.

p.s. Raw selenium isn’t the best tool for scraping unless you know it well. Scraping things that try to resist scraping might be troublesome, consider better tools. And for easy targets perhaps something like https://archivebox.io/

3

Bit ambitious, yes, and obviously not scalable, but directed or federated maybe.

4

Thanks for that link, looks interesting.. For selenium this Tor-browser git looks cool,

But quite old :frowning:

5

If you want scalable something like YaCy might be a better choice. But why would you want scalable? Also I’m not very big on searching, YaCy’s searching algorithm seems kind of weak… Have you considered it?

6

Thanks for the reminder, I actually looked into YaCy quite a while ago for some project. Will have to jog my memory.

7

Now this looks promising:

More as a search proxy which is more applicable.

4get is a self-hosted proxy search engine. It does not replace Google completely. It does not crawl the whole web or build its own search index. Instead, it sits in the middle, asks other search engines for results, and shows them back in a cleaner, lighter page.

(At the moment im kind of zeroing in either 4get or SearXNG as a search proxyVM. )

2 Likes
8

Discussing this is perfectly fine on the forum but the pros and cons of various scraping tools are clearly off-topic.

9

I’ve created quite a few selenium and playwright scripts (although not preferring the latter one knowing the owner) and what I can tell is that currently there is no better way (or the way at all) to scrape dynamic content of the pages and that includes a lot of fine tuning that depends on the page/site itself. A lot of scrolling, waiting, scroll down, scroll up, scroll back to the top of the page, constant changes of htnl layouts, avoiding AWS WAF and similar bot controls etc…

I haven’t found a “universal” way that works for all sites.