Org design is a strategic lever. How you structure platform, AI/ML, and product engineering affects delivery speed, ownership, and innovation. Get it wrong and you create silos or duplication; get it right and you scale leverage. There's no single right answer; the right structure depends on your stage, strategy, and culture. The key is to be intentional and to revisit as conditions change.
Centralized vs. embedded
Central platform and AI teams provide consistency and depth; embedded specialists accelerate product teams. Many orgs use a hybrid: central platform and model teams with embedded ML engineers or "AI champions" in product. Clarity of contract and RACI is critical. Document who owns what: who builds and runs the training pipeline, who owns inference and APIs, who integrates into product, and who is accountable for quality and safety. Without that clarity, you'll get finger-pointing and duplicated effort.
Choose the hybrid that fits your scale. Early on, a small central team with strong product partnerships may be enough. As you scale, you may need dedicated platform squads (e.g., ML platform, data platform, reliability) and clearer interfaces. Avoid the trap of centralizing everything "for consistency" and then starving product of velocity, or the opposite: embedding everyone and losing coherence. Balance is the goal.
Ownership and interfaces
Define clear boundaries: who owns model training, inference, feature store, and product integration? Use platform-as-product thinking: SLAs, roadmaps, and feedback loops. Avoid "platform as order-taker" or "product as bypass." Platform teams should have a roadmap that reflects company strategy and customer (internal) needs; they shouldn't just react to tickets. Product teams should consume platform through defined interfaces and contribute feedback and requirements, not go around the platform when it's slow.
Establish governance that keeps the system healthy. Regular syncs between platform and product leadership, clear escalation paths, and shared metrics (e.g., time to integrate, platform adoption, reliability) help. So does a simple RACI for new initiatives: who's responsible, accountable, consulted, and informed? Update it as you add new capabilities or teams.
Scaling the model
As you grow, consider domain-aligned platform squads (e.g., reliability, data, AI) and product-aligned teams that consume platform services. Keep the platform org small enough to stay cohesive and aligned with company strategy. A common mistake is growing platform in lockstep with product, which leads to a large, fragmented platform org that's hard to align. Prefer fewer, higher-leverage teams with clear missions over many small teams that overlap.
Revisit org design at least annually or when strategy or scale shifts significantly. Ask: are the boundaries still right? Is ownership clear? Are we optimizing for the right outcomes? Involve the people who live in the structure; they often have the best view of what's working and what's not. Org design is never one-and-done; treat it as a product that you iterate on.
Org design is never one-and-done. Revisit as strategy and scale change; optimize for clarity, ownership, and speed.