Thoughts on Open Source North 2025: LLMs, Agents, and Diagrams

I spent the day at Open Source North, and it was exactly the sort of conference experience I love: real ideas, real people, and those moments when you see someone’s mind click.

One of the most down-to-earth talks was about LLM evaluation. The analogy of treating LLMs like a smart but clueless intern is spot on. They’re bright, they’re eager, but they don’t know your workflows, your quirks, or your compliance requirements. The best approach? Give them tests. Real, domain-specific tests, like a certification exam. It’s not fancy – but it’s a strong option for building trust in these models. And honestly, that’s been my experience too: simple tests and real-world examples cut through the hype.

Agentic AI was everywhere this year – that sense that these autonomous agents are going to change everything. There’s some real promise there. Agents as teammates, proactive systems that actually do things, not just answer questions. But it’s also early days. Nick Roseth's talk was full of caveats: complexity, trust, cost, and that uneasy feeling of whether we’re just trading one kind of complexity for another. Still, I can see the spark there. If it’s real, it’s going to be huge.

The AWS and Pinecone talk on diagrams was one of my personal favorites. It didn’t have the flash of the agent hype cycle, but it felt grounded. AI turning messy, real-world diagrams into more messy diagrams? I've been there. They showed how to make it actually good! That’s the kind of tooling I want to see more of. The idea that architecture is more than just lines and boxes – it’s communication. It’s how we share ideas. That really resonated. If only I could generate a diagram of my AWS cloud in us-east-1 every time a commit merges. Infrastructure as code and presto, right?

One of my favorite parts of the day was talking with folks at lunch. There’s something magical about that moment when someone sees how these tools can be used to learn – like, really learning a new skill. One vendor was stunned when I said you can use ChatGPT to learn screenwriting, to get instant feedback, to learn faster than any traditional class alone. It's not a replacement, it's an augment. It’s that click moment where that nudge causes them to start asking the right questions. The moment is pretty cool.

We’re at the start of something big – but it’s not inevitable. The tools are real, but the magic comes from how we use them, how we supervise them, and how we make them part of our own journey as humans, not just as code, features, sprints, etc.

I enjoyed recapping with a post. Glad I could get this out same day. I've been to Open Source North a few times, notably some of the earliest when sdg first started it. A one-day conference still? It feels like this could have a second day soon. Fun to be at University of St. Thomas after so many years. It's enchanting to be on a beautiful campus in the late spring. I walked through the arches on the way in, for old times.