This is really cool. Do you mind sharing a bit about your process for coming up with the scoring system? Like, what was the process like when you were working with the AI to develop it?

Also, have you ever considered the ability to provide the site with, say, a name and some links to social media or a blog or any other websites associated with that person, and it provides a score on demand?

Thanks! At first I was using OpenAI's deep research to just give a summary and overall score 1-10, but I realized that could not be iterative and future proof as new evidence comes to light.

So after some thought, I switched to a system of individual evidence gathering and weighting each piece of evidence. I've given the models some basic starting points for types of evidence (for instance a donation has a default weight of 8/10), but have given the models leeway to make relative judgements.

After all evidence is collected, the weights and confidence that the evidence is accurate (usually very high) are put into a formula to derive a final score. No recency bias. The nitty gritty:

-Each row contributes direction × weight × confidence × status_factor, where disputed is cut in half and there is no recency decay.

-All signed contributions are summed into S, and total support mass goes into M. Final score is 50 + 50 * (S / (M + 4)), clamped to 0-100.

-That +4 prior mass keeps thin but unanimous evidence from producing extreme scores too easily.

-Neutral evidence (direction = 0) doesn’t push the score up or down, but it does increase M, which pulls the result back toward 50.

As for the ladder - I think that is a good idea, but in a controlled manner because of the token cost and potential for abuse.

This is really cool, but I'm wondering how you're getting around content moderation that the models utilize, or are you not?

I've definitely had the models refuse to process some content, like a lockpicking guide PDF. It happens pretty rarely, but I'm not testing with especially edgy content.

The agent is model-agnostic though; I already have integrations with 3 providers. It could be extended to more free speech friendly models in the future, should there be good ones that can handle tool calls well enough.

I recently watched a video talking about how much censorship there is currently with everything on the internet and how massive web pages and databases of information are just completely evaporating. I am not personally one of those "everything is burning to the ground" people, but it's getting hard to not at least see where those people are coming from, given recent events. Honestly, if you have the space available, getting this is probably not the worst idea ever.

Thank you for sharing

I'm not a developer at all, but is using something like Voyage and ChromaDB able to get most of the way there? I've implemented those for semantic search in a project I'm working on, but I'm wondering if rebuilding based on this would be better.

I'm not certain I'm the right market for this, but it's really cool! Is the idea basically to lower token utilization by piggybacking on someone else that is doing a thing as well?

You can do that by going to github or HN /new