Transcript
So if you part of a larger team, or interested in scaling your team, then Matt is the perfect person to be taking you through this next talk succeeding at scale.
Matt. Take your way. Cool. Thank you. Yeah. Really hard to follow that last talk. Probably one of my favorite favorite ones of the day, to be fair.
Good. Cool. Yeah, we can get kicked off.
So we're gonna talk about succeeding at scale today.
So bit of an intro to me.
Yeah, you can see him work for a gear set, of course. Been a gear set for about four years. Originally spent a lot of time on customer from patients. So, really diving into a DevOps transformation with the team and then spending time with them after, they become a customer.
And, yeah, privileged to work with hundreds of teams over that time. Recently moved to work on the product side, work with them in a different way, still focusing on those larger teams. So, this is particularly on workflows around CST pipelines or, larger teams and how that, that changes their workflow you go.
So cool. So intros, aside here's what I'll be talking about today. So first, I'm gonna explore the reasons why adopting DevOps processes on Salesforce can be challenged. We'll we'll come with some top level reasons. I'm sure there's there's hundreds more as well. Then we'll have a look at the difference team size makes as well and why this can be more of a challenge as that team size creases.
Then we're gonna have a look, some of the tips that we've used to help these teams succeed when they adopt processes, along with some of the examples of some of the functionality we developed, which is helping a bunch of teams today. We're gonna have a look at some next steps well. So what do teams do when they they get into a good state as well. So cool.
We get started. So to kick off why Salesforce DevOps is challenging. At a high level, there's these three key factors always at play. So DevOps process is primarily about organizing the folks in your team.
And this can be tricky to do on Salesforce because, you know, there's there's such a spectrum of technical backgrounds across the team, but there are no code admin. You you still off with absolutely no development experience. You're a pro code developer, maybe full stack before and come on to Salesforce, or you're somewhere in the middle, you're an admin developer.
And you had to do both. And there's there's all these other, roles as well that come into play, especially with larger teams, where you get the technical release managers, business users at play, when you get into UAT, architects, etcetera. Everyone has slightly different requirements and needs, and this technical diversity poses a a bit of a challenge when you when you get into that that team size.
Secondly, we all know Salesforce is a platform, traditionally focused around clicks not codes. So the developer centric features have grown, a little bit more slowly around the platform than you would in more traditional software platforms. So it it comes with a number of its own which means you need a bespoke set of tools for the job.
And, lastly, the the cultural shift, so whilst DevOps has been around for more than a decade, and even longer in terms of the underlying practices, it's still fairly new in the Salesforce ecosystem. Over the last few years, it's really started to gain action.
So the cultural aspects to DevOps and the working practice involved, you know, in the last talk change approval boards and things like that. Sort of these practices are are still ones that, are around in teams, and and some of the devil's practices are still relatively unfamiliar across Salesforce teams as well.
So we can start with the technical diversity challenge.
On a big picture level, your DevOps process, which should be a clear process for developing, deploying, and shipping changes.
It's gotta inclusive.
It has to work for your whole team. You don't want to end up in silos, and creating bottlenecks around the least part of the row, the least robust part of your process.
If you have some siloed team members who are still making changes directly in prod, deploying org to org or manually back syncing between orgs, then you're going to suffer slow manual releases, and you're gonna tread on each other's toes as well, which is the worst part.
So we know there's this spectrum of, technical backgrounds, and I'm sure some of you can see where you picture, where, like, picture yourselves where you, you sit on this as well. So it leads bit of a split experience on the Salesforce platform where different folks have different approaches to creating changes.
So some folks on your team may have different levels of familiarity with source control and DevOps in general. And the biggest challenge we have with teams, when, you know, you have a bunch of developers who are really used in practices using git, and bringing those admins on board, and and trying to bring them up to that tape technical capability is one of the biggest challenges.
The different groups across the team will end up working in silos, whether by role, what part of the platform you're building, what project you're working on, maybe the sprint team you're in, and this gets harder the larger the team gets because these silos tend to get bigger as the more work you're doing. So with small teams, it's really easy to maintain this consistent level of knowledge and technical capability. Keep everyone on the same page, share between each other, but the the the team gets larger, the number of folks at both ends of the scale starts to increase.
So as a result, you tend to lack a common and clear release process between each other.
So the second issue is around the technological barriers on the Salesforce platform. There's there's quite a few of these so you won't get through everything, but Salesforce is a a really good user driven platform, but it's traditionally hasn't really been built for collaboration especially on the dev side. It's really difficult to see who's working on which items and features. It's common to step on other people's toes and overwrite people's changes almost by design sometimes.
And for example, when changing or deploying really large monolith text mail files, like layouts, profiles, or just because you have a bunch of different dev orgs that's really hard to keep up to date with each other as well. So often this is gonna merge conflicts are, arising, and they exacerbate over time as, as this drift happens.
And finally, there's also complex dependencies in metadata and API quirks and limitations. If anyone's ever used to change that, or or ever used the metadata API, you will have come across some some some sort of over time as well.
At the cultural side, So I guess DevOps has been around for over a decade because it works.
Teams using these processes work better together and move faster.
And one reason teams adopt DevOps process is to really improve communication and break down some of the silos that you start C forming. So if you have development ops teams and release managers who don't communicate with each other, it's going to introduce some delays in your process.
If you have this proverbial fence that still exists where you kind of throw changes over the top of it for someone else to release that they weren't totally involved with building, you know, when when this is improved, teams release faster and more reliably.
This works best with a more lightweight in team approvals, and a shared responsibility. And we heard some if some folks were in the breakfast earlier, the Zurich team talking about how they bought kind of change approval boards into their into their team, rather than having sort of this external team who you always have to go through as gait and having a shared responsibility of those changes. So if you can move as much of these processes into your team and as early as possible and shift these issues left, you're not gonna get surprises on release day, and you're gonna keep the knowledge within the team as well.
So we're looking for a workflow which moves from separate silos, all working in different sandboxes like this, an independent work by each role in the team.
To something much more inclusive where all team members are involved in the process. Everyone knows where stories are progressed to. How they interact with other people's stories. And on top of that, having much better and closer communication across the team.
So these challenges are far from being unique to large teams.
They exist in teams of all sizes, when you a dead ops process, but as this team size increases, they bite a lot more and they compound over time. So for a team of thirty, ensuring everyone communicate actively so that folks know what their colleagues are working on is a lot harder than it is for a team of three.
For a start back, team size will usually be organized into multiple logical teams in the business, and a single sprint team isn't isn't probably going to contain more than thirty people So this immediately introduces these silos of information.
As a result, there's gonna be changes being made by folks on their user stories, which others on the team aren't aware of or others within other pods in that team as well. And on Salesforce, there's gonna be more of a challenge as well because you've got so many cross objects, so profiles, record types, layouts, which can impact otherwise unrelated work, worked on by others in the team.
So as a result, you're going to get conflicts that you'd see in git involving these objects. They'll come up a lot more when Lee's team size grows. That's even before you're considering kind of genuine conflicts between objects being made, in the meta atrial Apex as well. So as this team structure increases, there's gonna be a division of roles too that we mentioned earlier, and this becomes a lot more pronounced. Work with a lot of teams of three or four people where everyone has responsibility for their changes all the way to to prod, but I've seen very, very few twenty and teams for whom the same. By that scale, you've usually got a dedicated release manager, an architect, DevOps engineer, whatever the the, the moniker might be. Who manages the process of promoting changes through UAT and production.
With developers usually pushing their own changes up until that point. So those larger teams, the volume of changes being pushed through can lead to changes in the release schedule as well. So most teams have worked with more than team members are releasing on set cadence rather than just pushing features on demand.
And this can be due to a lot of factors, you know, end users probably want to know what's being released on a predictable schedule as well. There may be an a change approval board for example who, you know, there's a there's a set cadence you need to do that on, and maybe there's just some plain inertia in the team. You know, it's always the way we done things, so that's that's how we it now.
So I guess how can teams with this side often spread across multiple sprint teams in the company be successful in implementing a CICD process.
So the strategy we put out here for helping teams of all sizes set for success applies across the board it's gonna be more important as the team size grows. So number one is assess the starting point, know who you want to be in the process, what their roles are and where they're happiest working as well. So this is really going back to some users are going to be living in VS Code or version control system and be very, very comfortable with that. Others are going to have only experience of, a clicks based GUI to understand and push changes, and others are going to require a top down view whole process, you know, the release manager, the architect, etcetera.
Two, I think this is a common theme we've heard from a few folks today, but don't move too far too fast.
You don't need to do a big bang of everything just to start getting some improvements. So you don't have anything in version control and you're using change sets going org to org, moving straight to a fully fledged release pipeline in one shot, it's gonna be pretty hard to do successfully. So start to define some of the steps you want to take in in the long term.
Number three, start adopting some tenets of Delab's culture. So increasing transparency across the team, breaking down some of the silos and and and having good communication, and creating shared owner of the core processes as well. So the onus is more on the the individuals who are building things together, to take responsibility together for moving them on and doing it for high quality.
Shifting some of these processes left as well to be caught early in the process, so you don't get to UAT or prod, and start seeing issues there as well.
And for four and five, I guess once you've adopted these steps, is vital don't stop. Elite teams are always iterating on their process and measuring some of these improvements. So you can understand how they're improving over time and a lot of this measurement around DevOps performance team productivity, especially in the last year started to come up as a topic. So understanding things like the lead time for changes, success rate for deployments, how much what's our failure rate, for example, and what's our return on some of the investments we were making in projects is going to be really key to success as well.
Say, to make the best best decisions, you need to know where your staff from. We usually do this through, like, a DevOps maturity spectrum. I've seen Salesforce have one of these. I've seen many of our partners have their own maturity spectrum. So, all of them kind of apply.
It's kind of, as as teams move along this section from left to right there. Their process is gonna enable some faster deployments. They have some shorter lead times, lower failure rates as well. And if things do go wrong that can start to recover as well. But just looking at this, you're not really gonna know where you sit. You don't know whether you're a bike you're you're a rocket or a car, so it's better to actually quantify a couple of things as well.
So really good way to do this is through door and metrics, developed by Google to monitor success of their demo processes.
And as teams move from change sets to org to org deployments, start adopting version control, and then start relying on it as their source of truth. The cycle of releases shortens or should do.
So, deployments which may have previously taken weeks of planning, a possible on demand or several times a day.
Time taken for a story to go from a developer's initial pull request into production can be measured in hours.
And recovery time from system issues in in minutes as well. And the success rate of deployment should skyrocket as well.
If your team's still at those initial stages though of adopting DevOps processes, again, trying to move everything to the the right hand side, in one shot is unlikely to be a smooth journey, so picking out those metrics in your team that you could think you can move the needle on and what the step would be, makes more sense doing an iterative approach.
Cool. So best thing to do is figure out where you're starting from, who you need to be involved in the process and how they want to work as well.
So this is something where, something where some like a a pipeline UI would come in. This is making sure everyone's comfortable working in a particular place.
Admins or users less familiar with Gitworkflows or terminology, they're gonna prefer a guru, which effectively translates functionality to steps they need take to progress an item. Whilst developers are probably gonna prefer to still work in VS codes, get check-in, CLI, whatever they like as well. So they should be able do that in tandem as well.
So yeah, pipelines becomes kind of translation for this, helping users on the team have a common view of what's happening at different of the process all the way from those first commits and check ins through to production.
So second step is starting small and scaling up.
And the state of sales was DevOps report, that we've spoken about today as well. There's only nine percent of elite teams do, DevOps with a big bang approach. So it's it's important to realize that a big bang approach or doing everything in one step comes with a pretty big risk of failure. So DevOps involves a ton of different variables, whether that's team based, technology, you're based, and it's better to use an iterative approach to implement DevOps.
So your organization can focus on sort of continuum movements rather than trying to do everything at once.
So as we're moving through that maturity scale, we add version control as your source of truth as a pinnacle and the ultimate goal.
And it's a great point to aim for, and everyone, you know, in that case, you should just put everything into version control and start pushing changes through when it's not exactly what would happen on Salesforce. That meta layer destruction means some meta layer type just unsuitable for keeping under version control. The source of truth will always be production. For example, some unretrievable via the API.
Others like dashboards and reports, they can be environment specific. You you you don't want your testing dashboards to be pushed straight into action. So starting to actually choose the metadata you want to version carefully and starting off with a core set can be a good way to go. So what we typically see teams that start out with version control is maybe twenty five to fifty metadata types of very core config, and code as well.
And this covers the vast majority of changes that they want to push through and you can start to build on this over time.
There's a couple of different ways you can actually deploy changes between environments automatically. We recommend for some teams for performance based over APIs, looking at something like a a delta or commit by commit deployment.
But way you're confident that only the changes that you've you've committed or updated throughout the org, but again, there's different approaches. Some people like to do a complete. This is the branch, and that's going into York. So it's up to you. Cool.
The next piece as well is don't feel like you need to have all the possible environments in the process from day one or have a really complex grandchildren instructure, adding new environments and branches and sandboxes into your process should be easy later on, so don't complicate the process where you don't need to. Later down the line, you can just create branches from master. You can link environments to them, add them into the process if you need to.
So starting small also applies to you as an individual contribute feature as well. So when you're, creating feature branches, committing changes into them, you wanna keep features small as possible. You want to minimize the chance of hitting merge conflicts. You don't want these huge sort of, huge combined units of work moving up together. Reduce risk, ship small and often.
For many metadata types including layouts and record types, we're something like gear set, you can deploy just the parts of the objects that you'd want to, which which solves some of the issues for those huge files that you would see. So it's easy to avoid the conflicts in first place, but Apex code is another case where it's good practice reduce the size of classes being used. The smaller the classes, the less likely it is that multiple users are gonna be hitting the same objects.
So arguably the most important stage, adopting principles of a DevOps culture and shifting left. And this is summed up nicely in this quote. I think we always use this quote. But if you're able to test early, get code in the first environments, get it under source control and understand conflicts and issues at the very first stage. The chance of having items failed later in the development cycle is going to be dramatically reduced So it keeps the onus on those folks who built the thing again. You're not sort of throwing these things over the fence for someone else to fix and make sure that they they have responsibility to to solve the issues in moving forward as well.
So, there are many next to how Salesforce teams can achieve this from improving visibility of changes across the team, dealing with merge conflicts and keeping environments in sync through integration with project management tools like, like Jira, and defining a release schedule that works for your team. And these are some of the primary areas where the functionality, that we built comes into health teams, adopt some of these practices.
And I'll share a couple of the tooling and workflows we've enabled for these teams as well as some of the pit fours. I've seen them encounter over the last couple of years as well.
So the first piece is visibility the team. It's important for everyone in the team to have a common view of the state of the world. So across your team, your release manager, who's gonna need a high level visualization of all changes in flight, include reading the merge status, conflict resolution, validation status checks, work item tracking, all those good things.
And every member of the team needs to see where their stories are and kind of in the format that makes sense to them as well, and how they interact with the work being done by their team as well.
This is where something like, yeah, pipeline's UI comes in. Giving me that view of well. Stories are all parts of the process accessible to all team members. As well.
Next, we should probably talk about merge conflicts.
They're not an inherently bad thing, but they do stop you from overwriting someone else's work without asking them and making sure you're working constructively together as well.
It's gonna be really important flag up these genuine merge conflicts that can be resolved in the process and that way users can talk with each other early and resolve them before they become a much bigger problem later on close to production.
So having having some sort of, UI that people can understand to resolve these conflicts together makes sense but again, there's still inherent problems when folks, from an admin background who are coming into version control for the first time and looking at conflicts in an ex L file is even, you know, it's it's something to bring the team along for. So finding a way to easily do this together makes sense.
A bunch of conflicts as well that you normally get get a down to git not understanding the structure of XML so well. So it might see a conflict between two elements which actually were unrelated to each other. So, we put a semantic merge algorithm to deal with that to sort of take away or not even highlight many of these conflicts that you would actually see in your provider.
So looking for something that has Salesforce specific knowledge can really help reduce a lot of the load on the team as well.
This one's a perennial issue.
Really key cause of problems in the release processes environment drift.
It's not totally hard to stop, but if there are a load of changes in your main branch or production environment, which aren't in your lower environments, maybe hotfixes, have been back ported successfully.
You're gonna start to get problems when things are going through integration UAT.
New new features are often gonna hit conflicts and validation errors when you try and merge the bin. And to deal with this, we recommend merging, merging back from master or main into your lower environment frequently to make the whole process cleaner, and ensure these are well aligned.
Some of the key cases where we've seen people get into trouble is if they let those environments drift further and further out alignment, and and don't sort of get get ahead of it and solve it. And it's gonna cause more friction as they go through subsequent releases and misjudged more.
The next one's on the same thread. This becomes an issue if there are loads of changes that have been abandoned over a long period of time. So you've got, sort of, picky UAT testing or something that requested some extra changes and they weren't fixed forward or or reverted out. And so you've got a lot of changes that kind of stuck in UAT integration.
They've become landmines, really. They're they're a new source of between environments that has the potential issue to cause conflicts further down the road. And these conflicts as well, it's more likely you have to resolve them in like environment specific way when you get more further drift as well. So identify these features, revert them out, get back to state where you're aligned with production as well.
Integration with project management tools like Jira, is your DevOps So allowing information to be surfaced about these changes to the wider business.
There are two areas that we focused on with this. So just make the tickets are actually updated with all the information. There are some good links even if you're not using gear set between GitHub and Jira. So anything you can do to link those systems that be using to either manage work or, you know, commit work as well.
The other one is just making sure that we update the status of that story as you move through environments, and this can be keyed. So you don't have to attest and manually check with you that something's actually in UAT. They know that when a ticket is place. It's been automatically updated with where it's been deployed as well. I guess really key is you have, you know, tens of poor requests going through a day these teams. Cool.
And finally it becomes releasing to production as well. So you want to know what you want to achieve here. So different teams have different requirements for this step that we mentioned earlier. So we said feature by feature release is further along the Devox maturity scale and you know releasing when ready is the holy grail, but it's just not viable for most large teams.
And this can be for a load of different reasons. It's that users want to know when changes are hitting prod as we said, or the requirements of those business processes, or you want to release out of hours, for example, And at the same time, you could be working on a longer term project that's completely adjacent to your business' usual work, and that has to be done on a different release cadence in general. So a recommendation here is to find the process that works best for your team. Make sure that you can promote feature by feature for hotfix and other things, or build up a release over time, validate it continuously to production, and make sure it's integrated with those other hot fixes that are going out.
You should still be able to allow these individual features be deployed, and not have to wait to release it as well.
So how does this impact teams? When team members have a clear understanding of this process to get changes approved for implementation, it typically drives high performance across the board.
Ninety eight percent of teams from the reports show ROI from adopting Salesforce Devorts practices. It's not not surprising.
So when you've got all these processes in place, You shorten your release cycle, features are moving into production quicker than ever before with a higher success rate. What's next?
So teams of any size, but particularly large teams can't rest on their laurels. They're gonna have to be reassessing.
You wanna be learning iterating on process to ensure efficiency. I think there's loads of people who have spoken about where they've implemented DevOps today, but there's always this next step that they still have. So to make sure you're still identifying those and finding the new places can win as well.
And with this, there's several areas you can focus on. You can take tasks that are manual today and continue to automate them to to reduce operation or risk.
You can make sure you're testing what you need to test on every org at each stage. You heard UI testing come up a couple of times today on people wanting to implement that. Maybe backing up strategically important orgs and actually testing your recovery strategy, monitoring changes just to find out whether people are even jumping around the process as well.
So finally, keep measuring as you go because you can't really tell the impact of anything you changed unless you're measuring it.
The benchmark for this is going to be door metrics, covering release frequency, lead time, success, and recovery rates. And then within those, you can start to extract more data that could interesting for your team to improve performance as well.
And using something like our reporting API is a good way to go through it. So extracting some of this data, maybe putting it into Tableau that you could see, make it visible to the whole team so that they start to work on this as well.
So I'm gonna leave with a really cliche message. It's a it's a and not sprint.
Note what you want to achieve. Do it step by step. Don't don't go too far too fast. And you'll see some of these improvements your processes as well. But whilst you're doing this, just make sure you're keeping the team happy with these changes.
It's gonna be really critical.
That's pretty much it. If you want to go and find out a little bit more, we have a bit of an interactive workshop around CICD pipeline, so some of the function from the talk today.
Bassayo at the back will be running this workshop and I'll be around there so you can jump on some of the laptop yeah, hopefully you see somebody there and I see some some friendly faces. Yeah. Thanks.
