Description
Founder and President of MuseLab, Jason Lantz, highlights that DevOps ought to be a product-wide concern. He highlights that DevOps is core to your entire DevOps strategy — end-user experience and value delivery — not solely your development and release process.
This session was held at DevOps Dreamin’ London ‘22, the largest Salesforce DevOps event.
Learn more:
Transcript
I'm really excited to talk to you today about, sort of a, a how to take a more holistic view of DevOps that doesn't just treat DevOps as a dev problem, but really treats it as a product wide problem. And I think a lot of the themes in this session parallel with a lot of what we heard earlier. Who am I? I'm Jason Lance, and, I am currently the founder and president of UseLab, which is a company that I just formed in May after, leaving Salesforce where I was at for eight and a half years.
I've spent the last nine years of my life in this weird, I don't know, it's the Wild West or the Twilight Zone world of ISV style DevOps on the platform, which, you know, a lot of the focus in in what we talk about with DevOps is about org development model and how you maintain a single org. The challenges of an ISV building products that distribute to hundreds of thousands of orgs is a different type of DevOps challenge. So, somehow I fell into that interesting world. I was, in twenty thirteen, I was hired as one of the first two hires on the product team at salesforce dot org.
And was tasked from the beginning with figuring out the ISV style DevOps so that we would be able to scale to build out all the products that we built and everything. In the process over that time, I wound up building up a team of eleven people who are now, the Salesforce industries product delivery engineering team, at Salesforce.
Great, amazing team of people. And, they're continuing to work on the open source software that we built in order to be able to run our whole process there. And I wanna talk today, even though I'm a tooling developer and, really fascinated by tools, I really want to focus a little bit more on process and strategy. Quick agenda.
I want to talk briefly about ISV style software development and what I mean by that. Then I wanna introduce, I think, the challenge, which is, what I refer to as the DX silo effect. And then introduce a more holistic view or a solution to that of looking at kind of redefining it as the development to delivery experience, taking a holistic view of the product life cycle. And then finally talk about some of the challenges and kind of going from point a to point b.
My goals for this session, I'm hoping that, you come away with, are first off, that you'll consider using ISB style development even if you're not an ISV. There are a lot of use cases where it applies, and I think it's a really good idea. The second is to take a more holistic view of DevOps. Don't view it just as a development and a test problem.
It's a product problem, and think about it in that way. And finally, if you don't already, learn to love to use scratch orgs because I love scratch orgs.
ISV style product development.
I used to refer to this as just ISV product development but I started adding the style here because ISV style product development is basically using the ISV machinery of the platform to build products in an isolated way that then get deployed out to multiple orgs. So earlier, there was, talk about the multi org challenge, that people have when you have multiple org deployment of Salesforce, you needed common code.
So ISV style product development, differentiators in it, it's product thinking versus project thinking. There's a lot of really great literature about that. I won't kind of dive in here, but you can go Google and find some great articles on the difference between those. A characteristic on the Salesforce platform is you're using packages.
Packages are where you're distributing the core part of the product that you're building. And finally, they're designed for delivery out into a bunch of different orgs that often you don't even know what's in those orgs. So it's a different type of delivery model than change sets. In fact, you can't use change sets for ISV style development at all.
So who is it for? Obviously for ISVs, obviously for PDOs who build products for ISVs. But the three that I think a lot of people, a lot in the community don't consider are multi org deployments. So I know on the dot org side, UNICEF has one deployment per country.
They have one centrally managed package that they build and kind of distribute out to all of those. And then also, if you have achieved full digital transformation, your entire company, every department is running on Salesforce, each one of those departments is headed by somebody very politically powerful. And as discussions go on in the business, everybody's gonna want more autonomy and more control over the part of the Salesforce implementation that drives their work. Being able to give them more control, more autonomy by allowing them to develop an ISV style process that then eventually makes its way into your org development pipeline.
Finally, my favorite, open source. This community loves to share, but it's really hard to be able to share things in Salesforce without the right way of figuring out how to do that. Just making some metadata available in a GitHub repository isn't enough to build an open source community.
Alright. So let's talk about the the challenge, that I see, which is the the DX silo effect. And I think for ISV style development, the fundamental problem and how this silo effect comes about is because we often equate a product and a package as one in the same.
And in the experience that we had at salesforce dot org, it's actually not true. There are very few products on the AppExchange that you can install and immediately use.
There's a lot of post install configuration documents and things that you have to go through in order to set up. So while we're doing version control based development of the package source, everybody that's actually experiencing the product is doing it in a persistent org that has a whole bunch of non version controlled stuff set up in it.
So that leads to an org or a general team collaboration structure that's sort of like this diagram. Devs off on an island. They're working in scratch orgs that are very vanilla scratch orgs. They put things into version control, and then somebody takes that and puts it into the shared persistent orgs that are a bottleneck that everybody else is working in. That's where the product is first experienced.
So devs don't have the ability on their island to experience the product while they're doing development. They have to wait until it goes further down the pipeline. And everybody else is doing all their collaboration around those persistent orgs. The number of Slack conversations that I've been in of, like, this feature works in QA one but not in QA two.
And everybody has to scramble to try to figure out what's different between those orgs. That's a good indication of the challenges. So the impacts of the DX silo effect. The first is that if you define DevOps as something that is just a developer and test problem, and and it's just something to make your developers more efficient and catch more bugs, that's a lot harder to go get your business leadership to make a significant investment in.
If you're doing ISV style product development and your DevOps can expand beyond Dev and Test, now you're actually delivering product value rather than just internal team efficiency. The second is you can't shift left because you're locked into these persistent orgs. So you can't really utilize Scratch orgs to be able to shift left and test, before merge. Scratch orgs are generally only used by dev.
They're usually not even used by test because there's a lot of stuff that has to be manually set up in those scratch orgs to make it testable. Support, training, demos, all of these things require your enablement teams to go provision orgs. That org provisioning is really expensive at a manual operations perspective. And guess what?
The reason for, you know, all that pain that your internal teams are feeling every time they have to set up an org, your customers feel that too. So that's the the the challenge, I think, when we when we think about this as just a development and test problem. The redefinition that I've been sort of pondering on is this idea of talking about the development to delivery experience and taking a holistic view over the product life cycle. And so D2X combines or provides a holistic ISV product life cycle by combining the product delivery model with the tooling to implement it at scale. I mentioned a new phrase there, the product delivery model.
This is the most important concept, I think, to get out of anything and everything else that I talk about is all rooted in this.
It's a fundamental redefinition of what a product is. So the product delivery model says the job of a dev team is not to produce a package version as an artifact and then send that out, you know, toss it over the wall to enablement and the rest of the organization. The job of a dev team is to build an automation recipe and version control that can deliver complete experiences of the product into an org.
And so, and then that gets reused throughout the entire life cycle.
So a couple of key concepts related to that. That delivery automation is part of the product. It's not a trade off with feature development. I've had numerous arguments and debates with product managers who were like, well, I really need my devs working on features rather than automation. And in the product delivery model, that's a silly argument because it's the same thing.
The second is if there's missing delivery automation, that's a bug.
If you're a developer, you implement a new feature, and you have not automated the configuration so that QA can go get a test org that's immediately testable, you didn't complete implementing the feature.
And then finally, using the same automation throughout the product life cycle. So making the automation portable. It's not something that just runs in CI. It's not something that just runs when you spin up demo orgs. But you're trying to create this reusable automation.
When you do that, there's some kind of cool things that seemed previously impossible that become actually just reality. I've experienced all of these at scale and are really cool. First is you use Scratch Orgs for everything. There's no need for persistent orgs. The one exception to that is large data volume testing where you can't get a Scratch Org with lots of data storage. But aside from that, at dot org, we use Scratch Orgs for everything in the process.
The second is, you shift left and you fully test before you merge anything. So you don't have an integration branch where you merge a bunch of features together so you can get them into an org and test them and then find out that one or two of them was off. And then you have to scramble because you're all close to a release. You have to either roll that back or have your dev team scramble.
Nobody likes working in that environment. You can if you if you worked in an ISV before, you'll be familiar with this. You can regression test betas instead of having to cut a production version so that you can then get it into your test environment so that you can do testing, which is crazy. But it's a it's a constraint, that the platform imposes.
So you can also modularize your demo configurations.
So rather than having a one size fits all demo, for Source org, that has every demo story imaginable in it, You can have a baseline demo and then a bunch of industry vertical combinations that can be modularly added in to build up a tailored demo org that doesn't have to have everything in it. It has the story that you're after in it.
And, you can breeze through compliance audits because when you move everything about the delivery into version control, then you can use all of GitHub's built in compliance mechanisms to breeze through compliance audits. And finally, you drive down the implementation cost of the product.
So this is where it really goes beyond the internal team. You can make these recipes available so that when a customer goes to implement your product, it costs them less. That means that your product team can now sell to a mid market or the lower end of the market that they were priced out of before because of SI costs.
So these recipes to deliver product experiences are really key. And there are five layers that I see of things that you need to do to deliver a complete product experience into an org. So the bottom layer is dependencies. Just to point out, if you work in change sets, dependencies are something you never have to care about because it's just automatically there and they work magically for you.
But when you have to step away from change sets like in the ISV model, now you're responsible for dependencies. So if you build an extension package you have to have the underlying package in your scratch orgs before you can deploy any of the source that you're working in. So your recipes need to automate that. They also need to automate things like features, licenses, org settings that are required in order to deploy the package source or install the package.
Then, you have the package source, which is what most ISVs have in version control. It's basically the second layer. Then some might have some portion of, kind of unmanaged or unpackaged automation layer on top of that that manipulates the data out or the metadata outside of the package.
One of the cool things in in Cumulus CI is the ability to do metadata transformations. So you can define automation records or automations that are safe ways of doing something like adding a pick list value or adding a standard value set or adding a field to a page layout without knowing what the layout in your org is. So it's doing a basic ETL operation, retrieve, modify, deploy it back. And so you can build these really rich automations that allow you to deliver the metadata changes safely to an unknown org configuration.
And Then on top of that, you have settings and config. So custom settings, metadata types, config as data, all the OmniScript stuff that we just heard about, CPQ. It's an anti pattern to me, but it's definitely something that you have to manage in utilizing some of these products.
Now, I do want to point out there's org settings, in this list as well. And org settings is also called out in dependencies. The difference here is this is an org setting that is not required to have the package installed but it is required to create the experience that you're trying to build with this recipe. So at this point, you're talking about configuring the product for a particular experience.
And then on top of that, you have datasets.
I really recommend and I think it's a good best practice for teams to have one common dataset so that when UX is talking about personas, when a dev goes and spins up an org, the personas are there in the data set that's in the org. When a doc grader goes to spin up an org from a feature branch because they know that they're going to have to capture new screenshots, The data is already there. It's all the same data and everybody's working on the same data, working on the same story. When you're creating these recipes, you will probably create multiple recipes for a product. But I definitely recommend try to create as few recipes as possible.
Get as many people working on common recipes because you're getting people working on common stories, as you're doing that. And then the second is build these automation recipes however it is that you build them, in a way that is portable, so that you can kind of reapply them at different stages throughout the process. Really try to avoid making them tied to a particular infrastructure or something like that.
So when you make this change to the development to delivery experience, the structure of team collaboration changes pretty fundamentally.
Collaboration moves away from that handful of persistent orgs into version control and human to human interaction. Self-service scratch work creation drastically cuts the operations burden that somebody on your team is feeling now. It might be devs.
If you have a team called DevOps, I'll bet that they probably spend over half of their time manually provisioning orgs and configuring them for other people to hand off. That goes away because the recipe to completely create an immediately usable Scratch org is what Dev is building in version control. And the entire team is enabled on the tools to allow them to create a Scratch org on demand from any feature branch that they're working on or, you know, that's in development. So if, you know, your UX designer wants to go look at the implementation of a particular feature that they did the design for, they can go spin up a scratch org from that feature branch and click around it.
And they don't need to go ping the developer to spin up the org for them. So I want to focus mostly on the process, but I do want to point out, kind of, the tooling stack that we built up because some of these pieces, I think, filled gaps that make this process a little bit more palatable. So at the bottom, obviously, is the Salesforce platform that provides a lot of different APIs. It provides orgs, packaging machinery.
Salesforce DX came along and provided source format conversion to GP scratch orgs.
Cumulus CI sits on top of that as an orchestration framework.
The way I always like to describe this is, Cumulus CI competes with bash.
Rather than writing a bunch of bash shell scripts that call a bunch of SFDX commands, you can orchestrate everything in Cumulus CI using a YAML configuration in the repository.
There's a universal configuration that it comes with. So there's one common experience in working with any Cumulus CI project. And then each project in their GitHub repository customizes that. So there's all these out of the box recipes that have the same name across every project. But each project customizes them to what they need. And then you can have client applications that utilize that portable automation that's defined in the repo. So CCI is the command line interface that's mostly used by the internal team.
GitHub actions can also run that same automation for CI, use cases. And then one of the other tools in Cumulus suite, is Metadeploy, which is what powers install dot salesforce dot org.
And Metadeploy essentially allows you to take these automation recipes and publish them out so that users can connect to their org and run the full sequence of automation against their org. And all of these tools are open source. They're all currently maintained by the Salesforce Industries Product Delivery Engineering team, my former team there, and continually being enhanced.
So, quick time check. Cool.
I wanna briefly sort of show what a development process looks like with this. This is actually broken down pieces of a much bigger North Star, diagram that I built and shared out on social media, and I've got a link in the in the resources. Key thing to point out about the development review process is all development is done in Scratch Orgs. All development is isolated in feature branches. And what's being developed in the feature branch is not just the package source, it's the recipe to deliver a complete experience of the product. Every commit, that gets pushed up to GitHub kicks off CI builds. There's a series of CI builds, that we recommend.
Unmanaged. So deploy the unmanaged, package source, run Apex tests. Then we also automate creating a two g p version of the package, setting the information about that as a commit status in GitHub so it's available to anybody else on the team.
And then we run tests against the package version of the code as well. In the test and review phase, whoever wants to get involved in reviewing this pull request before it merges, whether it's UX, maybe the dockgrader wants to get involved, product manager. Definitely QA should go in and take a look at it and test it. They all get one single common flow. It's called QA underscore org underscore two g p. That same flow run-in any Cumulus AI project will get you an immediately usable org of the commit that you checked out built from a two g a two g p test version of the package.
One other thing to point out.
We operate generally on this idea. The branching model is really simple. Main branch is the release train. And I like to think of the release train more as a subway.
It's the the train car that's sitting there at the station and features when they're, you know, when they're ready, they run down, they get on the train. The one thing that messes up the subway well, one of the things that messes up the subway system is when a sick passenger gets on the train. So the whole goal of the pull request here is shift all testing left, test everything until it is release ready, and only merge it when it's ready for release so that you don't put a sick passenger on the train. Because when you look on the release side, the goal of the main branch is to always be ready to cut a version and, deliver it out to people.
So whenever a new PR gets merged in, there's automation around building a new beta version of the package. There's automation around spinning up a scratch org, installing the beta, running the Apex test, and running browser tests against it. And then one difference in SFDX is we use GitHub to persist release information.
So if you've created packages in SFDX and you have to go edit the SFDX product dot JSON to record, Okay, we created this new version and we're rubbing something, you never have to touch any of those files. You never have to commit after creating a release to record that you created a release. To me, it's this tricky situation because you rebuild that commit that recorded that you created the release. Anyhow, the artifacts are created as tags and releases in the GitHub repository. So you can inspect the repository to find out the packaging information.
And regression testing is done in a scratch org. There's a built in regressionorg flow in Cumulus CI that will install the latest production version and then upgrade to the latest beta. So it's actually simulating a customer upgrade from the current production to that latest beta.
The reason betas can't be tested in persistent orgs is because once you install a beta into an org, you can't ever upgrade it. So if you're relying on a single org and you install a beta into it, that org's gone. Now you've got to rebuild the org. Or you have to use some other mechanism like use a TSO and spin off clones of it or something.
But it's clunky. If you can automate creating a complete regression org environment in a scratch org that becomes much easier. And then when it comes to actually cutting the release, the release manager goes and kicks off a build. It's manually initiated, but then one hundred percent automated as far as all the delivery pipeline.
So when you think about delivering for ISV style products, there's kind of three different pieces of our, delivery infrastructure that we had built. The first is, I guess just standard ISV delivery infrastructure. So you can schedule push upgrades of the packages. I'd recommend that you have your push upgrades be scheduled from your pipeline through automation.
And also, the pattern worked really well for us of push upgrading to all customer sandboxes. And then after a period of time, for us it was a week, to all customer production orgs so that they get some time to be able to test out that new functionality with their customizations in their org.
And then the other piece of the standard ISB machinery is Trialforce Source Orgs or TSOs.
This is a persistent Salesforce org you can go in and make changes in, create a template ID off of, and then you plug that into the Trialforce machinery. So when somebody wants to come get a trial of your, product, that's what it's based on. For most ISVs, that's just a persistent org that somebody manually goes into and manually makes changes. In this process, that's actually folded into the path to production, which means that it's covered by all the compliance enforcement in GitHub.
Nobody touches the TSO directly. The pipeline runs recipes against the TSO to update it in the way that it needs to be updated. So you actually have version control base tracking, over that whole process.
The second kind of area of deployment is, this part of Cumulus suite called meta deploy, that I mentioned earlier. And meta deploy allows you to publish the recipes that you've created and that you've used internally out so that customers can be able to run them or your demo teams can be able to run them. So all of our installers for the products at Salesforce dot org run through install. Salesforce dot org.
And a good example, this is actually my favorite example of the product delivery model. I apologize. I know that's really small. I put a link to it, off to the side.
But this is the grants management product for salesforce dot org. This product requires two it's an extension of two managed packages. It requires some unmanaged metadata to be in the org before you can install grants management.
Then after you install grants management, there's a whole sequence of unmanaged metadata modifications that normally would be in a post install doc. Those are all automated here. It's creating an Experience Cloud site. It's configuring the site. It's publishing it for you. It's loading a set of flows that automate a grant making process. So when you're a customer and you go through and run this, at the end of it, you have a pretty full product experience set up in your org for you automatically.
It's a lot more than just installing the package version, which is what you get now if you try to install from the AppExchange.
And then finally, you've got all these recipes that are really valuable and can be mixed and matched and are modular. We didn't totally adopt this as much as I would like, but I do think that this is an idea that every ISV should consider, which is take whatever automation you have to allow you to build environments of your product and create an open source repository that has that automation in it so that your partners, your customers, can actually tap into all of the recipes that you've been using and that you've been building to make your own internal environments, Especially if you want to have ISVs extend your product, it's really helpful to them to give them all the recipes that you made to develop your own product.
Briefly getting there, it's a journey. My services practice that I'm in the process of building up is like a six month engagement to sort of guide through this this whole process. But it generally starts with discovery. And I think the easiest question to ask in discovery is, how long would it take you to rebuild your TSO?
Or how long would it take you to rebuild your QA org? What are all the steps that you would have to do? Because that's your tech debt. That's the gap that you need to figure out how to automate.
And I think once you do that, then you really need to work on buy in. I think, there was discussion in the Centers of Excellence session earlier about the importance of getting buy in, not only from leadership but also from the engineering team. You and and I I found the best way to do that is to focus on the pain points that everybody's feeling, and show how this new process can do that. The next is enablement.
Train up the team, train the whole team. Don't train one person and have them be the tool expert. This is part of the product development. It's part of the product experience and everybody on the product team should know how to use, at the very least, how to create an org from version control.
And then that initial recipe that you build automate the QA org configuration so that you can replace your persistent QA org with scratch orgs. And that allows you to test a feature before you ever have to merge it, which is great.
And then when you look to kind of roll this out, I think start with the internal dev team, fix some of those challenges, really replace that persistent QA org with scratch orgs that are done through automation, then roll out the release operations process so that you streamline release operations. And Then start looking at how you can improve the delivery. How do you better enable whoever's doing implementations whether it's an in house services team or you have implementation partners how do you enable them to utilize all this automation that you've built up as part of the product? This will be shared out, I presume.
So there's links here to everything. I started a blog about a month ago where I've been writing about a lot of these concepts. I tried to kind of prioritize those here on the right. Two things to call out.
There's a trail on Trailhead called build applications with cumulus ci. If this sounds interesting, if you haven't taken it, go take it. If you are remotely familiar with SFDX, it should take you about a half a day to get through. And it's a great overview of, kind of, you know, a lot of what I've been talking about.
And the last thing is, at the top right, a link to my Calendly. I love talking about this stuff. Even if you totally disagree with me I was a debater for four years in high school and two years in college I want to debate with you. I want to hear about the experiences that you've had that I haven't.
Because this is the Wild West. We're kind of collectively figuring out what these best practices are, and, I don't have all the answers. I don't have all the experience. So thank you.
