Patrick argues that GenAI technology, much like cloud before it, needs a dedicated platform team to scale effectively across an organization. Drawing on his experience as a VP of Engineering at Showpad, where he helped bring AI features to market, he outlines a three-part framework: platform infrastructure, enablement, and governance. The talk was presented at ETLS, introduced by Gene Kim who describes Patrick as “the Godfather of DevOps.”
The platform infrastructure component covers a wide range of shared services that individual feature teams should not have to build independently. These include centralized model access and governance, vector databases for indexing unstructured data, RAG-as-a-service pipelines, unified API proxies for controlling access and routing, traceability and debugging tools for prompt chains, and production health checks for LLM functionality. Patrick emphasizes that just as monitoring and observability became shared infrastructure in the DevOps world, the same pattern applies to GenAI operations.
On the enablement side, Patrick stresses the importance of helping teams overcome their fear of working with AI. He recommends providing sandbox environments for prototyping, encouraging experimentation with local models, and being pragmatic about framework adoption. He highlights that testing GenAI applications is a major pain point for developers, since the output is non-deterministic. He walks through different testing strategies: exact matching, pattern testing, sentiment analysis with helper models, semantic distance checks, and using an LLM as a judge to evaluate another LLM’s output. Patrick notes that 2024 feels like the year the industry is finally taking evaluations and observability seriously.
Patrick also reflects on how the developer role is shifting from producing code to reviewing and managing AI-generated output, drawing a parallel to the “ironies of automation” paper from the early DevOps era. He warns that as developers produce less code themselves, their ability to review it may atrophy, and suggests that the time saved through automation will likely be reinvested in learning, resilience engineering, and chaos engineering for AI systems. On governance, he covers topics such as desktop-recording AI tools, opt-out policies, licensing concerns, and risk levels, urging organizations to educate their teams on these safety considerations as they scale GenAI adoption.
Watch on YouTube — available on the jedi4ever channel
This summary was generated using AI based on the auto-generated transcript.