Register or log in to access this video
New York • September 8 & 9, 2027
Loved LDX3 New York? Pre-sale tickets for 2027 are now available.
One of the core goals of platform engineering is to reduce developer toil and accelerate delivery across the entire development lifecycle. While many organizations focus these efforts on CI/CD and tooling, other platform surfaces such as service connectivity and networking often remain fragmented and owned by individual teams. This fragmentation creates significant hidden friction that is rarely visible in traditional productivity metrics.
At Databricks, product teams building services across the control plane and data plane routinely spent multiple weeks setting up basic connectivity. Engineers had to configure proxies, manage DNS entries, provision load balancers, and reason about environment-specific constraints such as intra-cluster, inter-cluster, and cross-region communication. Despite extensive documentation and shared standards, this work remained slow, error-prone, and highly dependent on senior engineers. The result was persistent “shadow toil” that distracted teams from delivering product value.
In this talk, I will share how our platform engineering team made a deliberate investment to simplify service-to-service connectivity by introducing uniform platform abstractions and out-of-the-box discovery. Working cross-functionally with networking, compute, service framework, and authentication teams, we replaced bespoke, team-owned solutions with a consistent connectivity model that worked across deployment environments. I will walk through the developer pain that motivated this decision, why documentation and standards alone were insufficient, the architectural and organizational trade-offs involved, and how we rolled this out incrementally without breaking existing systems. The results included reducing service onboarding time from weeks to minutes and achieving over 70% latency improvements on certain inter-cluster paths. While this talk focuses on service-to-service connectivity, the lessons apply broadly to other platform domains where inconsistent abstractions quietly slow teams down as organizations scale.
Key takeaways:
- Why investing in platform engineering can deliver disproportionate gains in developer productivity
- How consistent platform abstractions simplify architecture while improving latency, cost, and maintainability
- How to identify and quantify “shadow toil” in areas like networking that often escape traditional metrics
- How a developer-first mindset helped align multiple teams around a shared platform abstraction
