Pycon.de keynote: machine learning models in a dynamic environment - Isabel Drost-Fromm¶

Tags: pycon, python

(One of my summaries of the 2025 pycon.de conference in Darmstadt, DE).

When the web started, you had a few pages. After a while you couldn’t remember all the URLs anymore, so curated overview pages cropped up. Then even more pages, so the first automated search engines started appearing. And thus “keyword stuffing” as search engines only looked at the manual keywords. So search engines started looking at links between pages and sites. So people started gaming that system, too…

Same with email. With email came spam. And thus automated mail filtering. And spammers adjusting to it. And spam filters adjusting in turn. And on and on. A cat and mouse game. Not everyone in the audience remembered this cat and mouse game with search engines and spam.

If you have a security mechanism, you can expect the mechanism being attacked. A virus scanner can be used to attack the system it protects…

She once saw a quote from Harold Innes, 1952: “it should be clear that improvements in communication tends to divide mankind”. For example the invention of the printing press. Soon afterwards you had someone named Luther and a split in the church and some wars, for instance…

If you’re evil, which service to you target?

• Popularity with target audience.

• Ease of influence.

• More user features.

What helps you as an evil person?

• Anything that looks natural.

• AI anyone? Text, images. Helps a lot.

• Business metrics.

• Intransparency of the site/platform you’re attacking.

LLM training. What are you training on? On crawled web pages? Pages that nowadays are filled with LLM-generated content? Bias upon bias, perhaps? There was a project called “wordfreq” that contained word frequencies. It stopped recently as “post-2021 language isn’t usable anymore”.

OODA loop: observe, orient, decide, act. A quote by Elen Friedman, 2007: “the ability to look implies the ability to influence”.

As said at the beginning: not everyone in the audience remembered this cat and mouse game with search engines and spam. So watch out with LLMs and AI training: there is a cat and mouse game here, too. Watch out what you’re doing yourself. Observe what your users are doing. Observe the ways bad actors might influence your training. Orient yourself, be aware of your surroundings.

Photo explanation: picture from our 2024 vacation around Kassel (DE)