In our “Engineering Energizers” Q&A series, we shine a spotlight on the groundbreaking engineers at Salesforce. Today, we feature Jing Vergara, Software Architect, who developed the internal release platform that powers Salesforce’s intricate Hyperforce infrastructure. This platform enables global teams to deploy code safely and at scale.
Discover how Jing’s team created a unified deployment strategy for thousands of diverse services and scaled release workflows to handle over 250,000 weekly changes, translating to approximately three changes per second. At this scale, trust and safety must be embedded into every release, making Managed Releases (MR) the cornerstone of secure, reliable, and high-velocity software delivery.
What is your team’s mission?
Our mission is to enable safe, scalable, and efficient deployments of production changes across all Hyperforce environments. At the heart of this mission lies MR, a powerful platform that seamlessly moves code from development to production, embedded with robust safety protocols to minimize risk and ensure trust.
MR is more than just a CI/CD tool. This Salesforce-built platform is tailored to the unique complexities of Hyperforce, enforcing safe change practices through automated gates and replacing manual oversight with automated evidence gathering and validation. For instance, MR automatically verifies that the code has enough code coverage and ensures that deployments follow a strict environment lifecycle: dev → test → stage → prod. Additionally, MR restricts changes to off-peak release windows when necessary for critical changes, implementing a “follow the moon” model. This approach allows for releases during non peak hours in one region, followed by observation and validation during peak time before expanding globally.
Success in deployment is only part of the equation. MR ensures that services remain healthy and performant after changes before broad rollout. By integrating these safeguards into the deployment lifecycle, MR shifts the foundation of trust from tribal knowledge to automated, reliable processes.
What’s been the hardest challenge in orchestrating safe code deployments across Hyperforce environments?
One of the most significant challenges involved designing a standard rollout strategy to safely support Salesforce’s extensive service diversity. Hyperforce encompasses over 2,000 production services, each with distinct architecture, code branching management, operational characteristics, fleet size, regional distribution, public cloud presence, deployment strategies, and risk profiles. Coordinating a reliable release process across such a diverse landscape is not a simple task—this is a complex distributed systems problem compounded by organizational intricacies.
To address the complexities, the team created a reputation risk scoring system that determines the rollout order. Rather than treating all environments equally, the system prioritizes order of deployments based on criticality. Lower-risk instances receive changes first, enabling early detection of anomalies and minimizing the potential blast radius impact. This tiered approach is going to evolve into dynamic scheduling logic that considers traffic patterns, ensuring changes are implemented during periods of low usage and battle-tested across geographical regions as they hit the peak periods.
Salesforce’s platform sees over 250,000 code changes deployed to production weekly.
How do you scale release workflows to support 2,000+ services and 250,000 weekly changes?
From the outset, scalability was a critical design consideration. We realized that Spinnaker, an open source deployment solution that we originally started with, couldn’t keep up with Hyperforce’s increasing complexity. When we developed MR, we prioritized both deployment scale and service diversity.
Some teams manage a single microservice with a single Docker image to a Kubernetes cluster, while others, like Marketing Cloud, deploy hundreds of Amazon Machine Images (AMIs) per release. This wide range of needs makes standardization particularly challenging. Moreover, other teams operate with fleets of over 10,000 instances, each requiring consistent rollouts of security patches, feature changes, and configuration updates—often on different schedules.
To tackle this, we architected MR to coordinate multiple services and release cadences simultaneously. The platform supports layered release strategies (e.g., canary, staggered groups, and global for emergency break-fixes), version management, and real-time validation gates. When different types of changes overlap—such as a performance fix and a security update—we are improving MR’s safety logic to ensure the superseded version is not accidentally applied.
It’s not just about increasing the number of deployments; it’s about orchestrating them at an unprecedented scale.
How do you balance rapid deployment speed with trust and stability in a multi-cloud environment?
The solution lies in our engineering culture and architectural discipline. From the start, we viewed MR not just as a product but as a blueprint for safe and reliable changes at Salesforce. We established a set of architectural and operational guidelines that every team member must adhere to. These guidelines include comprehensive design reviews, early demo scripting to ensure usability, and multiple layers of safety checks across code, configuration, and deployment processes.
Moreover, the MR team uses the platform to deploy MR itself. This self-hosting practice allows us to catch bugs early and ensures that every improvement is rigorously tested under real-world conditions.
The results are impressive: Since 2021, the MR platform has had only one or two production incidents, despite managing tens of thousands of releases. Trust isn’t something we add later; it’s the core of our approach to every release.
How does your team ensure that updates in one area don’t introduce regressions elsewhere across Hyperforce’s release infrastructure?
One of the biggest challenges with MR, given its components are maintained by multiple teams, is maintaining consistency and coherence as features develop. To tackle this, we’ve implemented a rigorous cross-team review process. Any significant changes, whether they involve new features or architectural overhauls, go through structured reviews that involve both internal team members and stakeholders from related platforms. These reviews are not just formalities; they include detailed impact assessments, failure scenario simulations, and customer-focused demo walkthroughs.
We also participate in broader architectural reviews, bringing together architects from various teams. These reviews address topics pertinent to Hyperforce, including developer/release experience, impact to the Hyperforce platform, and availability considerations. This external perspective provides us with the confidence that our enhancements won’t cause unexpected issues for all engineers across the Hyperforce layers.
Furthermore, we emphasize a shared testing responsibility. Each team member is accountable for verifying their part of the system, but we also conduct comprehensive integration and user journey testing to ensure the entire platform functions seamlessly. This blend of individual ownership and collective accountability is what prevents regressions from making it into production.
Jing describes the culture within Salesforce Engineering.
What innovations are shaping the future of automated software releases across Hyperforce environments?
Two primary areas are driving the future of MR: extensibility and automated remediation.
First, extensibility is essential because the range of Hyperforce services is expanding, along with the number of substrates/cloud providers we support. Each team and environment has distinct compliance, deployment, and safety verification needs. MR must adapt without causing delays or duplicating efforts. To meet this challenge, we are developing a more modular architecture with standardized extension points. This allows other teams to add their own policy logic, hooks, or validation workflows without affecting the core platform.
Second, we are delving into AI and machine learning to speed up detection, diagnosis and recovery from failed deployments. Currently, when an issue arises, engineers have to investigate, diagnose, and take action—often under tight deadlines. Our goal is to create AI-assisted systems that can automatically categorize failure types, pinpoint root causes, and either initiate an automated rollback or guide the necessary corrective actions. These processes will be data-driven, leveraging real-time telemetry and incident pattern recognition, rather than relying on fixed rules.
Together, these R&D initiatives aim to evolve MR from a simple release gatekeeper into a dynamic, goal-seeking, self-healing, and self-improving system for managing changes.
Learn more
- Stay connected — join our Talent Community!
- Check out our Technology and Product teams to learn how you can get involved.
