feat: added third blog post

content/posts/harvester/index.md

@@ -0,0 +1,261 @@
++++
+date = '2026-08-12T22:00:00+02:00'
+draft = false
+summary = "... for when you're a puppy, not a politician <small>/ref</small>"
+title = 'The "private" metasearch engine that sends user data to a knockoff Palantir Foundry instance'
+tags = ['investigations', 'privacy', 'fediverse']
++++
+
+### Introduction
+
+**hi and welcome back!!! :3** this time we will research & discuss a ridiculous violation of the privacy of possibly thousands of unknowing entities, happening through the websites `tiago[.]zip` and `estrogen[.]delivery`.
+
+I'll start with this right away: **please please please please do not harass Tiago. he's just a kid.** He's also seemingly not doing this in bad faith or with inherently malicious intent. Not like that makes it okay, his actions are still very wrong, but please (as the reader) be on your best behavior.
+
+This blog post will be split into two sections, detailing two issues I and [Nyx](https://nyx.girlkisserwx.com) found over a little over one day. The first is the [indiscriminate, wholesale scraping](#scraping-the-fediverse) of the entire fediverse, without anyone's permission or consent, and with the few documented methods to opt-out [not working](#-and-you-cant-even-opt-out).
+
+This led to us discovering the [indiscriminate fingerprinting](#harvesting-metasearches) of everyone who uses any instance of Tiago's metasearch engine, which he expressly claims doesn't happen in the README, and sending telemetry (including exact searches, translations, and device information) to an instance of some analytics tool, that's [made to look like a fake Palantir Foundry instance](#pawlantir).
+
+## Scraping the fediverse
+
+I first became aware of this dataset existing on August 2, 2026, when a close friend I knew for a few years found my fediverse account from (mostly) just the string `sel`, using Tiago's ["mastodon.parquet" dataset](https://tiago.zip/datasets). At the time, I didn't really look into this, because I was on vacation and had other matters to attend to.
+
+Note that anyone with an internet connection, a web browser or terminal, and a pulse can download this dataset for free and instantly, without any authentication at all. Through the website, one does need to provide an email address and a *"brief description of what you want to do with [the dataset]"*, in order to download it. This information is simply logged somewhere and then the user is instantly sent to the download URL, which as said, is totally unauthenticated.
+
+Nyx asked Tiago about this, and he stated that this is lack of authentication is intentional, because otherwise *"you wouldn't be able to have a script download them"* <small>(couldn't you give the script a bearer token or something?)</small>. He also stated the email & reason requirement is there because *"most people won't go into the source code to check the actual urls"*, and that *"it's mostly just a record"*.
+
+### What's included in this dataset?
+
+The `profiles.parquet` table as of writing has the following columns:
+
+| column            | type                     |
+| ----------------- | ------------------------ |
+| `id`              | varchar                  |
+| `acct`            | varchar                  |
+| `locked`          | utinyint                 |
+| `discoverable`    | utinyint                 |
+| `created_at`      | varchar                  |
+| `url`             | varchar                  |
+| `avatar`          | varchar                  |
+| `followers_count` | integer                  |
+| `statuses_count`  | integer                  |
+| `indexed_at`      | varchar                  |
+| `emojis`          | varchar                  |
+| `username`        | varchar                  |
+| `display_name`    | varchar                  |
+| `bot`             | utinyint                 |
+| `is_group`        | utinyint                 |
+| `note`            | varchar                  |
+| `uri`             | varchar                  |
+| `header`          | varchar                  |
+| `following_count` | integer                  |
+| `last_status_at`  | varchar                  |
+| `fields`          | varchar                  |
+| `inserted_at`     | timestamp with time zone |
+
+The `instances.parquet` table as of writing has the following columns:
+
+| column             | type                     |
+| ------------------ | ------------------------ |
+| domain             | varchar                  |
+| users              | integer                  |
+| title              | varchar                  |
+| first_seen         | varchar                  |
+| last_seen          | varchar                  |
+| status             | varchar                  |
+| inserted_at        | timestamp with time zone |
+| software           | varchar                  |
+| version            | varchar                  |
+| description        | varchar                  |
+| admin              | varchar                  |
+| languages          | varchar                  |
+| active_month       | integer                  |
+| open_registrations | utinyint                 |
+
+The latest `posts.parquet` table (2026-08) as of writing has the following columns:
+
+| column                   | type                     |
+| ------------------------ | ------------------------ |
+| `id`                     | varchar                  |
+| `created_at`             | varchar                  |
+| `account_id`             | varchar                  |
+| `content`                | varchar                  |
+| `uri`                    | varchar                  |
+| `language`               | varchar                  |
+| `visibility`             | varchar                  |
+| `sensitive`              | utinyint                 |
+| `in_reply_to_id`         | varchar                  |
+| `in_reply_to_account_id` | varchar                  |
+| `replies_count`          | integer                  |
+| `reblogs_count`          | integer                  |
+| `favourites_count`       | integer                  |
+| `reblog_id`              | varchar                  |
+| `quote_id`               | varchar                  |
+| `media_attachments`      | varchar                  |
+| `poll`                   | varchar                  |
+| `inserted_at`            | timestamp with time zone |
+ 
+### "i won't ask for permission cuz you'd say no"
+
+Back to the timeline; on August 12, I decided to look into this a little more. After exploring around on `tiago[.]zip` a little, I found [the faq page for the scraper](https://tiago.zip/almanac), which explains how to opt-out <small>(we'll get to that in a minute)</small>, and why this mass-scraping isn't opt-in to begin with:
+
+> because almost nobody opts in, and a snapshot of the few hundred people who did isn't useful for anything. it also wouldn't reflect the network, which defeats the point of studying it.
+
+This is a paternalistic and almost self-defeating justification. The problem with "I won't ask for consent because nobody would consent" is that while you yourself are assuming that no one would consent, you're proceeding regardless <small>(because "the ends justify the means" i guess??)</small>. Microsoft usually does this kind of consent-laundering bullshit with hostile defaults, "Remind me in 3 days" and similar.
+
+update: as I was writing this, it was [updated](https://bookstodon.com/@keith/117088937719512629) with:
+
+> if you don't know about it you can't opt in, and a snapshot of the few hundred people who did isn't useful for anything. [...]
+
+### ... and you can't even opt-out
+
+note: things in this section might be outdated by the time you read this; this represents the state of affairs as of writing this, on august 12th/13th 2026
+
+The same page mentions 3 ways one can opt out; 1 at the user level and 2 at the instance level:
+
+- add `#noindex` to one's bio,
+- enable authorized fetch on one's instance,
+- or disable the public timeline on one's instance.
+
+We'll discuss the last two first. I was pretty surprised after reading those, since my instance has had authorized fetch enabled since the second it went online, and public preview of anything has also been disabled for a long time. I figured the simplest explanation was that either the scraper was broken, or that it simply pulled in my instance & account from another instance.
+
+I ended up messaging Tiago about this, and he confirmed that the latter was the case. He also offered to (retroactively!) delete my instance and related data from existing datasets and also future releases, which is super nice of him, and I think confirms that he's not being malicious or acting in bad faith.
+
+<small>(yes, i did get permission to show this)</small>
+
+![A screenshot of my discord DMs with Tiago. I tell him about the issue with my instance being opted in, and that I am unhappy with his scraper. He replies saying that he will make the opt-out process better as described above, and offers to manually blocklist me.](dms1.png)
+
+![A screenshot of my discord DMs with Tiago. I tell him that I'd appriciate being manually blocklisted, and he retroactively removes my instance from all previous exports.](dms2.png)
+
+As for `#noindex`, if you query `note` of `profiles.parquet` for that exactly, you'll get zero results. However, with the query `SELECT COUNT(*) FROM 'profiles.parquet' WHERE note ILIKE '%#%noindex%';`, you will get 663 results (or 674 if you omit the hashtag, but those 11 people probably just don't know that they need to make it a hashtag):
+
+![Two queries in the 'profiles' parquet table, for the amount of users with bios including '#noindex' and just 'noindex'](ddb1.png)
+
+Nyx asked Tiago about this, and he said that it's because he accidentally made the tags case-sensitive (so `#NoIndex` wouldn't work, but `#noindex` would.) This may be true, but I ended up finding a better explanation. [Mastodon](https://github.com/mastodon/mastodon), [GoToSocial](https://gotosocial.org/), and possibly other types of fedi software put a single `<span>` between the `#` (hashtag) and the name of the hashtag (in our case `noindex`). 
+
+```html
+<a href="https://instance.tld/tags/noindex" class="mention hashtag" rel="nofollow noopener" target="_blank">
+    #
+    <span>noindex</span>
+</a>
+```
+
+(GoToSocial instance, profile is included in the dataset)
+
+![the error being present on GoToSocial](gts.png)
+
+(Mastodon instance, profile is included in the dataset)
+
+![the error being present on Mastodon](masto.png)
+
+(Sharkey instance, notice how there is nothing between the hashtag symbol and the name of the hashtag)
+
+![the error not being present on Sharkey](sharkey.png)
+
+(Iceshrimp.NET instance, ditto)
+
+![the error not being present on Iceshrimp.NET](isnet.png)
+
+This seemingly happens for most included profiles (not sure about why there's a difference of 2 here, but it's negligible):
+
+![A query in the 'profiles' parquet table, for the amount of users with bios including specifically 'noindex' split apart by an HTML span element, as described above.](ddb2.png)
+
+### Conclusion
+
+First of all, I really hope Tiago will fix the existing methods to opt out, but at least the `#noindex` one, which currently is confirmed to not work for 2 very popular types of fediverse software (those being Mastodon and GoToSocial).
+
+Tiago also told me in our DMs that the current methods to opt-out are *"very much temporary"*, and that he's working on *"something where you'd just type your profile url and get opted out"*. The problem with this is that this would require people to know they're being scraped and can opt-out, which most probably don't.
+
+Whether scraping most of the fediverse, without consent, and publicly releasing it wholesale is ethically justified or not (it isn't) is something everyone has their own opinion on, so please tear each other apart about that in the comments of wherever you arrived here from <small>/hj</small>.
+
+## Digging deeper
+
+Something that Nyx brought up was that `tiago[.]zip` was talking to a service named `palantir[.]estrogen[.]delivery`. If the reader doesn't know, Palantir is a data analysis company that provides services to (among others) militaries, governments, ICE, pigs (metaphorical), and pig farmers (literal), just to name a few. Some of their products include Palantir Gotham, Foundry, AIP, and Apollo.
+
+This will be explained in some more detail in a later section, but all you need to know for now is that **there is nothing Palantir-related on `palantir[.]estrogen[.]delivery`!** It's just a login page pretending to be a Foundry instance.
+
+### so where's the estrogen?
+
+Looking at `estrogen[.]delivery` (peak domain btw), it appears to be a domain for hosting various personal services. To log in, one uses their hrtID account ('hrt' is [hormone replacement therapy](https://diyhrt.info/)):
+
+![A split view of both the hrtID login screen (has a small sqaure image of a catgirl, and options to log in with a passkey, github, a password, or make an account), and the 'estrogen delivery' homepage (black screen with an image of cats, and links to 'id', 'mail', and Tiago's website)](hrtid.png)
+
+Some things hosted here appear to be, among others:
+
+- an authentication server (`id[.]estrogen[.]delivery)
+- an e-mail server (`mail[.]estrogen[.]delivery`)
+- an instance of a web analytics program (`palantir[.]estrogen[.]delivery)
+- another adjacent thing (wdk exactly) (`panopticon[.]estrogen[.]delivery`)
+- a seemingly now defunct archive of TommyInnit-related tweets (`tommyinnit[.]estrogen[.]delivery)
+- a blog that's probably an AI tarpit (`blog[.]estrogen[.]delivery)
+- some twitter app (`twitterbux[.]estrogen[.]delivery`)
+- a "friendly knowledge base for gender-affirming hormone therapy" (just use [transfem science](https://transfemscience.org/) bwaaa) (`wiki[.]estrogen[.]delivery`)
+
+![A graph of the wiki's pages. There's only 14 articles, most are about transfeminine HRT-related topics, and some are just nonsense (like "Those who know" or "Tuff").](wiki.png)
+
+We don't really care about these services, except for the Palantir one, which will be important in a second.
+
+Another thing that came up while digging through domain records was `yuri[.]delivery`, which just appears to be a site for looking at yuri. I don't know if this site is related to Tiago or not, but it is in the same Cloudflare DNS zone (so there's a chance that it's managed by the same account). Once again, it is irrelevant to this story.
+
+### Harvesting (meta)searches
+
+A thing that kept showing up in the Censys results for `estrogen[.]delivery` were a bunch of services named `search[.]example.com`. Now, if one looks at these, these are simply instances of [Tiago's metasearch engine](https://github.com/tiagozip/metasearch) hosted by random people, such as this one:
+
+![A simple search page featuring a Kasane Teto plushie, a search bar, and links below to 'self-host!', 'about', 'api', 'bangs', 'translate', and '\[cloudflare\] frankfurt'](search.png)
+
+If we inspect the network requests, we see two important things: the first is an obfuscated JavaScript file from `https://palantir.estrogen.delivery/Pn7xQk2.js`. A simple grep of the source code reveals that it happens on every page:
+
+![A grep of metasearch's source code for 'estrogen dot delivery'. Every HTML page has one result, which is a script tag in the head to the aforementioned JavaScript file.](grep.png)
+
+Following that, there are several network requests to `https://palantir.estrogen.delivery/`, starting out with this "pageview" one:
+
+![The same search page. There's a network request's payload selected, showing that some of our device telemetry as described has been sent to palantir dot estrogen dot delivery.](pageview.png)
+
+This gets our **language**, **color scheme**, **timezone**, and that we're a returning visitor. It also gets the exact page URL (this will be important in a second), and our **screen width** (but not height, for whatever reason). Let's try searching something:
+
+![A search query for "Why are wasps really cool actually" with a few results and videos. There's a network request's payload selected, showing that our entire query was sent to palantir dot estrogen dot delivery.](wasps.png)
+
+oh no. The request to the fake Foundry (fakedry?) instance includes our exact query. Both as a specific 'query' field and in the field for the URL.
+
+It also includes our screen width (a few times), and a bunch of web metrics (some errors, cumulative layout shift, first & largest contentful paint, interaction to next paint, time to first byte, and similar). This alone would be bad, but not even nearly as bad as including the search query. This entire thing goes against what Tiago himself wrote into the [README](https://github.com/tiagozip/metasearch#private-by-design):
+
+![A picture of the readme of metasearch: "private by design", "no tracking is used by default, and you can search the web with no cookies or accounts.", and "we do not log any type of data and your searches are never stored or analyzed."](readme.png)
+
+### ... and translations too
+
+The cherry on top is that same thing can be observed with the "translate" part of the website. Here, the URL contains the source & target language and the source text (notice how it included 3 separate versions of that too, as I was typing):
+
+![A translations page, showing "this is extremely saddening, by the way" being translated to portugal. There's a network request's payload selected, showing that our source & target language, and source text, were sent to palantir dot estrogen dot delivery.](translate.png)
+
+### Pawlantir
+
+While in the heat of the moment this didn't really occur to me, **this login page is fake**. There is nothing Palantir-related or made behind it. It looks really convincing though, [they even got the path to the Foundry login page right](https://platform.censys.io/search?q=%22%2Fmultipass%2Flogin%2Fall%22) (`/multipass/login/all`):
+
+![A login page that looks like it would be a Palantir product. There's a palantir logo, palantir icons for the email field, a "Remember me" checkmark, and a "Next" button.](login.png)
+
+Some obvious signs of it being fake are the title being "Super MAGA Palantir ICE Peter Thiel AI Data Harvester 9000 // Palantir Server", as well as the fact that trying to log in with an invalid email address does this:
+
+![The same page, zoomed in, with the text "KILLED" over it in scary red letters.](killed.png)
+
+By looking up one of the class names (`bp6-input`) on Censys combined with the path, we can find that this login page was taken from `palantir[.]4x[.]rip` (and slightly modified to add the "killed" thing):
+
+![A censys query for "bp6-input" and "/multipass/login/all", showing two results: palantir dot estrogen dot delivery, and palantir dot 4x dot rip.](censys.png)
+
+Which confirms it:
+
+![The visible page on palantir dot 4x dot rip as of writing this. It confirms that the login page is fake and is used a curtain/costume for a self-hosted analytics instance, and had to be retired due to a "recent legal correspondence", specifically a takedown request](joke.png)
+
+## Final thoughts & acknowledgements
+
+I'm not sure where to go from here with this. All of what I learnt here is pretty troubling and I'd love to hear it be addressed. I'm sure Tiago's a lovely individual (he was back when i knew him ~1.5 years ago :3), so I can't see why he would be doing all of this without even really trying to hide it. I'm left disappointed and a little confused.
+
+happy phone. <small>/ref</small>
+
+Thanks a lot to Nyx again for helping with the research and interviewing, to Tiago for his cooperation and for showing good faith, and to you for reading <3
+
+Some other sources and liveposts:
+
+- https://bookstodon.com/@keith/117085275218189650
+- https://old.mermaid.town/@futzle/117086174531531720
+- https://fedi.wasp.best/notes/aptptbsrdya9mskb
+- https://fedi.wasp.best/notes/apuy1uchy1aef5gf

themes/webbed-theme/assets/css/main.css

@@ -1,4 +1,4 @@
-@import url('https://fonts.googleapis.com/css2?family=DM+Sans:ital,opsz,wght@0,9..40,100..1000;1,9..40,100..1000&display=swap');
+@import url('https://fonts.googleapis.com/css2?family=DM+Sans:ital,opsz,wght@0,9..40,100..1000;1,9..40,100..1000&family=JetBrains+Mono:ital,wght@0,100..800;1,100..800&display=swap');
 
 :root {
   color-scheme: light dark;
@@ -194,6 +194,7 @@ small {
 img {
 	max-width: 100%;
 	border-radius: 0.25rem;
+	text-align: center;
 }
 
 table {
@@ -215,7 +216,24 @@ th {
 
 code {
   color: light-dark(#dc8a78, #f5e0dc);
-  font-family: 'Cascadia Code', monospace;
+  font-family: 'JetBrains Mono', monospace;
+  font-size: 18px;
+  padding: 0.25rem 0.50rem;
+  border-radius: 0.50rem;
+  
+  background-color: light-dark(var(--light-secondary), var(--dark-secondary));
+}
+
+pre {
+  overflow-x: auto;
+  white-space: pre;
+  padding: 1rem;
+  border-radius: 0.75rem;
+}
+
+pre > code {
+	padding: 0;
+	background-color: auto;
 }
 
 blockquote {