DevOps zero to hero: How we implemented DevOps at Payroc

Share with

Description

Join Matt Bevins (Senior Developer at Payroc) and Piotr Zuralski (Senior Product Manager at Gearset), where they discuss Payroc’s Salesforce DevOps journey from the beginning to the present day:

In this talk find out:

  • Payroc’s DevOps goals
  • How Payroc chose a Salesforce DevOps platform
  • How Payroc’s DevOps process works with Gearset
  • Top tips for teams looking to implement DevOps

Learn more:

Related videos:

Transcript

Very pleased right now to introduce you, to Matt Evans, senior developer at Payrock, and to my esteemed colleague, Piat, who is going to be talking about the DevOps journey at Payrock. Gentleman take it away. Thank you so much. Cool.

Thanks, Jake. Yeah. So welcome to our talk on, DevOps zero to hero and how we, meet our team at Payrock implemented DevOps.

Quick introduction between, middle for myself then, senior, a tech care senior engineer, but it's a developer role, being a payroll. I've just over two years now, before, yeah, I've got a couple of certifications behind me.

Probably see a lot at the London user group. So I'll go to the admin, devs. I'm one of the co leaders of the data tribe.

But I also try and get to as many as possible.

And an interesting fact that most people don't know is I'm a dog trainer. So I spend my Sundays, on a field, taking a class, but also training my two vehicles that I've got.

Matt, I'm Pilt. I'm a product manager at Giset. I've been at Giset for, nearly two and a half years. And, yeah, we've just been working with hundreds of teams.

To help them create and deliver stories faster and provide more business to value.

Quick disclaimer, we're going to do a sneak peek at turnip coming an upcoming feature, but please make any, purchasing decisions based on Jenny available software as you always, the motor is a dream dreamforce.

So what?

Could you say a few words about Payrock and your team? Yeah. So Payrock probably most people have not heard of the company. We're a fintech company who, provide payment solutions, for businesses, we offer things like credit card processing, point of systems, e commerce solutions, and various payment technologies to, our merchants, which are sort of the end shops and different bits like that, and agents who we go out and sell our services, and then partners who integrate with our systems, based or head offices based over in Chicago, in the USA, but we have a number of offices, across, the I've got, the UK and all the various different places in America.

We this I took this picture from the website at the bottom, it says eighty one billion, but we're probably more around the ninety billion plus transactions at the moment this year.

And a couple of brands that you may have heard of as well net payments, big in the gateways and sort of next gen is one of the brands in the UK that put in the credit card terminals in different bits like that.

About my team, so we're a quite small team. I will refer to us as a small team, six of us, as you can see on the screen, mainly responsible for Salesforce and an assistant called Agreement Express, which is, like DocuSign, it's an agreement signing tool.

We work in sprint.

So two week sprints currently, and but we don't we let wait till the end of the sprints release, we, so we're pretty agile in that we'll get things out as soon as as soon as they're ready and that's been signed off by the business.

So yeah. Thanks, Matt, for introducing Payrock and your team. Could you paint us a picture of what sells deployments look like before you implement it, that that website payroll?

Yeah. So I suppose we started about a year and a half ago, on this journey was that we, had a slightly different team, because what we have now, weren't just in the process of introducing sprints and formalizing our, cadences to get things out a little bit more quicker.

But we had shared sandboxes.

No source control or any tracking of changes, and any big release we had meant we pretty much span up a new sandbox.

And from there, we were sort of juggling, having to see what we hadn't released, changes that we had going through the pipeline, where there was dependencies, meant that we had to either include them in our deployments.

And, yeah, we, and or we would have to until whether a a sort of a situation where we could actually, get them released as part of a bigger piece of work work.

I suppose at the point, some points we were over icing each other's because of the shared sandboxes, if we'd someone else to sign up a new sandbox, because they needed to be a little bit more in sync with production, they could deploy like a page layout or something like that, which would over right someone else's change, but we also have the problems where someone has included a new field and a page layout that, obviously, the dependency problems, and you then had to include that field, which you may not have won in your releases.

So as you can imagine, we started stepping on each other's toes that came a bit of a nightmare.

Definitely.

So it's clear that your team, wasn't following a single process, and everyone was using different bits of tooling.

What sparked your search for a dev ops solution?

Yeah. So at Payrock, we have, initiatives, like, back then it was hackathons.

So twice a year, we would get together when we had the opportunity to doing what we would do, normal user stories day to day and look at something that we could add value to the business.

And so, I suppose the very first hack form we went, okay, we're not we think we can improve our dev ops process or dad how we deploy. And so we started on setting some goals, as you can see on the screen, things like, one process for all.

We as a, when I sort of mentioned one process for all, we had in, we're getting a student placement who was out of university. We hadn't touched, Salesforce. So we wanted a process that could be adopted by someone who didn't really know, Salesforce and things like, mezzo making it easy, from there.

And, version control, sort of the Jira integrations, different bits like that. And so once we've set these and everyone agreed to them, it was just probably as we're setting up the, what we were going to do as part of the hackathon, I had previously used geyser, or had some experience of it at my last company. And so I've recommended it to the team.

So we went out and or I've got in con putting a little quest for on the website, going to one of the sales team, Benny, he's here today. Doesn't appear to be in the bright yellow shirts. I don't know how he managed that, but he is around. Go bug him if you, what any questions about Gisa.

But, we got in touch with him, explained what our goals were and how we had three days for the hackathon.

To try and show a value from what a new DevOps process could, incorporate for us.

And, so we agreed, sort of the first day of the demo.

We got into it.

Had a little, look through the various, feature of gear set, because gear set has, it's been around for a while. There's a lot of features.

And from there, we'd quickly implemented pipelines, which was in beta at the time.

Some trial sound, some not trial sandbox. We've had Sandboxes set up for this little, hackathon, and the team started to use the tools to try and deploy the changes through the pipeline.

As you could imagine, a DevOps process, trying to go from zero, which pretty pretty much had to a fully blown DevOps process where we I pretty much had, at that point, we decided for every bit of metadata we had in our orgs into our Git repo.

Had a few problems along the way. So flow, things like the comparisons and the CI jobs just ran slower because it was having to do more.

So if I I suppose the conclusion after the three days was, yes, we've shown some value of you, implementing this, but it wasn't as good as, I I suppose I'd promised the team, or I was hoping to pro give the team, but all also, it we thought there could be something better out there.

Sorry, I suppose we showed the initial value of having a more formal process, but we decided, okay, gear set is better than we had, but there could be better tools out there. For sure. So it sounds like your initial, introduction to Geesett as part of three day hackathon didn't go quite as planned, but you persisted and you changed your approach, Could you take us through what you did next? Yeah.

So as, as we like to do at Payrock, we have bake off. So when we have a, a new product that we're also a new something new we want to juice, we don't just go out and go that's the that's the tool we're gonna get. We compare what tools are out there. So, as part of, the, obviously, we a little bit of knowledge we got from the three days working with gear set.

We decided to look what else was out there, so blue canvas for and Capardo, Salesforce were just releasing DevOps Center at the time, and that obviously many more out there, but we decided to focus on some of the bigger ones that, had gone out there.

Also looked at what metadata we thought we it in version control, tools like this, you don't always have to put sync in version control. You can't just deploy them, using, like, auto log deployments. So we sort of decided, okay, things like custom objects is a must. We want to track what changes we're going through on there. Apex classes flows, method custom metadata, email templates and things like that, were all things were sort of the big things for us at that time because we were in the process of onboarding more teams into Salesforce on the service cloud.

So we wanted to make sure we did that. But we also, to make it fair on when we were looking at the other tools, we decided we'll have some set tests.

So, we wanted to make sure we had simple tests, things like we wanted, to make sure that we could change a field, and things like the label of the field, the size of the field, the pick list values, things like that. Pretty standard stuff. But as part of our DevOps process, the permissions was quite a a thing we wanted to incorporate.

We wanna have to deploy and then have a load of manual steps of having to say this field needed to give this permission to all these different users. So we wanted to make sure the tool we selected could easily deploy things like permission sets, list views, flows, the standard sort of stuff, but the more I suppose other bits that you don't often think about is maybe destructive changes being able to remove the tech debt stuff. You don't want to leave it in there in your source control. So things like that. And we wanted to be able to keep this process in sync.

So where I sort of mentioned we had shared sandboxes, we wanted to give the ability to, our developers their own sandbox, so they didn't have to tread on each other's toes. And so having that process where we could make sure Evink is in sync, and we could see where a change was. Quite important as well. And then we also wanted to really, give the solutions a good test by looking for things like dependency term test. So if we deploy this flow, which references the field, a new field that the developers forgot to add in the, the metadata. We wanted to see if the tools would pick that up, things like, and also conflicts, so if two people, two of our developers were working on the same class or something like that, we didn't want them one just to overwrite each other. We wanted the conflict deletions come in as well.

And because we're working trying to be more agile and work more efficiently, we wanted then to be able to group smaller user stories into a bigger release.

So as part of that dev ops process, we Slide off with Blue Canvas.

And if you're new to DevOps, this tool is probably the simplest one to implement. Out of off the shelf anyway.

So, blue canvas is, you basically sign into it. You connect your orgs, and as part of you connecting that org, Blue Canvas will go out and sync every bit metadata to its own branch within their source control. So you don't need to worry about your own repo or anything like that. Blue Canvas sort of did everything about, for it yourself.

Things like, org branching strategy, so blue canvases had its own. So it was an org based, branching strategy, and because that's in their own repo. And when you came to do changes, it was a branch, the branch comparison So it was super quick to do be able to pick the metadata. You could check then cherry pick the me metadata because you could see exactly what had changed, but that also, had its sort of negative sides have to wait for Blue Canvas to sync, with that org to then see, that metadata changes.

Have got that down to pretty quickly now, but sometimes we've found that, the we had to clone deployments for it if you already started a deployment and realized that there's a bug in that I need to re redeploy something, you'd then have to clone it for it to then do that branch comparison again from from the system.

And one, I suppose, one of the other problems we had was lightning email templates were big on bringing the new team in at that top point.

So we needed to, as part of that team, be number to Service Cloud, we were trying to deploy, email templates to get so they could get up to speed quickly.

And we just couldn't do that in Blue Canvas for whatever reason. I must say these features, this was a year and a half ago. So might not be the same now. So if you want to go out and test, test it yourself, and see if it's still there, to emissions and permission sets, could be deployed, but it was they had a separate tool, which they sort of tried to incorporate into their product.

So it's you could do it, but they weren't properly grouped with, with the change that you were deploying.

So from there, we went on to Flayson, so Flayson, is like, a little bit like Capardo because it lives within a Salesforce org.

I think the difference I think you can do it with Capardo, you could, by default, I think Fosum will spin up a separate org so it doesn't live inside your production. So that was a positive because we didn't worry want our dev ops process to live inside our production org.

But, I suppose it was some a simple setup because it was just connecting the orgs again, and it would sync the changes. They seemed quite quick as well to pick up the changes.

So the with, flow some, their branching strategy, that we were testing was, you a story. So that user story that you build as part of the deployment, you pick the metadata and that user story would then just be deployed to multiple walks. They'll obviously see what conflict resolution, you sort of enclosed something that sort of had a load of buttons at the top that you would work your way along to deploy, and there was just a lot of buttons and, different bits like that.

So rollback was super simple, and there pipeline process at the time was more of a I'm gonna configure these number of steps.

And if it was successful deploying at one point of in that step, it would just then go on and deploy it to Nextorg and then Nextorg. So you can quite quickly, take make a change and deploy it quickly, through the various systems, but it wasn't something you could, have, I suppose that control of I want to deploy this in this org. You'd literally have to have a different pipeline process for different points of that.

But it did allow automatic deployments as part of that. So feedback from our team and And myself, obviously, we found the use of interface a little bit clunky because it was inside a Salesforce org, so they're limited to a Salesforce UI, and we know sometimes that's not the greatest.

Integration, integrations, so integrations to it third, pie systems like Jira.

Actually, you know, igrations like PMD, I suppose it's the one I was referencing with that comment, is was required manually setting up. They weren't, and in when we were doing it, we had to spin up a Heroku instance to actually let me get it to be able to scan the code that we were deploying.

So it was a little bit more clunky, required more manual steps in the setup process.

The Jira integration was there.

It was about the formatting that went back into Jira that added it which I did teams, would you look at, see where we were within our delivery cycle just wasn't formatted as very as nice. It didn't really give a nice feel. So that was, like, we weren't, peed on too much, but, the Microsoft team and we're a big Microsoft Teams user, and we're all, work remotely.

So teams getting when something's being deployed is sync we wanted, and there was no team's integration out of the box. So it would these have to sync. We've had to custom build.

So yeah, so on the back of that, so we saw that's two other systems we looked at.

We also did look at Capado very quickly, but obviously it's, quite an expensive solution. So, and it also to keep the cost down, you would deploy it into your production.

Or so it wasn't saying we really looked into much further after that.

Dev ops center, I said, was quite new at the time, did install it, but we found that once you got past the initial stage, of, like, once it got into after UAT, any changes that have been deployed into UAT, then got bundled into their package.

So you couldn't then say, I just want to deploy this change into production that might have changed since, but we, it was, yeah, something we, came, it wasn't as saying we wanted, we wanted that control of what we deployed into production at what point. So during this process, Benny, the sales guy, was still in contact with us at peer set. So on the for our conversations, we decided that let's give Giercer another go.

Taking on what we've learned before and, what so if you can see, what things we liked about Clearset was, obviously, it was a, a separate web app, and the user interface probably is the most intuitive, for us anyway, what we found to how do you deploy. It's, yeah, nice and simple.

Bought in feature branches, which would, obviously, we had, we've seen before, which obviously branch is normally linked to a user story, so you could easily, see where that user story was within the pipeline.

You have this is pipeline view, probably a little bit small on the slide, but, that's pipeline view, you can see these little numbers, by the orgs in there. And that shows what user stories are waiting to be released or what feature brought features of pull requests are waiting to be merged into them, orgs. So it's very clear to see what your, orgs, if they're missing anything, what they had still waiting to deploy, and it's easy to, obviously, deploy that stuff in.

Obviously, other things that were good you could still do walk to walk deployment. So anything we decided that weren't part of our source control and our continuous improvement pipeline, we could then also just use gear set to deploy and have the ability to roll back that change, even though it wasn't in that source control, integrations, so out of the with gear set, Jira is integration there.

Microsoft Team, Slack, Chatter all was very simple to set up. And the other things to note is what the team noticed was that you have, release notes that pop up in, in the corner of gear set. And so in the, So very quick to release new features, and you could literally get notified when something else has come about. And if we did have problems, we could quite quite a very quickly jump onto a live chat with a gear support team, and they would talk you through.

The problem and help you jump on a Zoom call if needed to a screen share. And when we were comparing some of the support with the other packages, gear set was the only one that had the live chat feature. All the rest was email and then wait for the support to come back.

So but we did still note, even with what we'd learned about comparisons, and we're only picking the metadata that we knew that we needed to select comparisons were still slower.

So off the back of, that the where we now where GESET did come our winner. This is a little table we put together in Confluence.

We're big advocates of pipelines, and as we've done, since we've just gone over our year, with gearset, we've done over nine thousand appointments over, thirty one year, probably about nine thousand five hundred deployments now since I bought these numbers.

See, keeping all the orgs in sync were over probably thirty two thousand now.

And you can see a number of comparisons run quite a lot. So yeah. Thanks so much for sharing your your experience. That was, what a kind of on, and shall I say a delicious idea to do a Davos bake off?

It's also amazing to see how well your team has adopted the gear sets and the dev ops and how many deployments in CIO brands you've done. Could you, talk a bit about kind of where you are with your dev opportunity at the moment and what's the next steps are for you and your team? Yeah.

So would say, our big I wouldn't say, our focus, we're already with dev ops process. We always try and improve upon. But what big thing that we're gonna be looking at next is, testing. We do a lot of peer review at the moment. So as, I'm the senior dev who, comes up with a design for a lot of our user stories, who there's a couple of other designers well, but whoever does the design, at the moment, we're doing the peer review of what the developers are built.

We that takes it away from doing what we need to do. So we want to try and, look at automating the testing, putting, the regressions test inside of it, build a test suite, with some automation and we're in talks with, some company at the moment, getting demos again. So quite possibly have another bake off, but it will be of the testing tools, but one of our things will be that we want it to integrate with gear cert. Because that's our deployment tool.

So we want to make sure that once it reaches a certain environment that the tests are automatically kicked off and it can't progress further until then tests have all been completed and don't introduce any bugs that we know of. Super exciting. Any any tips for any team, that wants to implement DevOps, embark on that journey? Yeah.

So, Obviously, one of the big tips I would have is start small.

We're a bit like deer in headlights with the hackathon.

Decide what metadata you want in there, look at what solutions available, And I will say, look at free versus paid because the free solution is an always the quickest to market.

And Rob at the back is gonna be a French touch dreaming. I think he's got a talk on exactly, free versus pay tools. So you can always speak to him at the end, and you never know if you might get him at one of the user groups that do the same, time and effort is, is a big thing with free versus paid.

I've had experience where we try to implement our open source at my old company and ended up wrapping it and then go down a paid version.

Definitely trial them.

There is no, I would say there isn't one dev op solution that suits everybody, though, that you can even out of the box, some of them you'll have to adapt, maybe some of your processes too. But don't settle on just the first system you come across and make sure the system has, room for growth, things like Capeado, if we didn't know Capado, Blue Canvas was a very quick tool to get up and running with. But we felt we would have hit a point where we couldn't improve our DevOS process further.

Adoption is key as well, to bring it's not just your team adoption. It's bringing your other teams on within the business, and make ensure they know because, nor, sometimes, when you implement a new DevOps process, your cadence deployments may go down, but, your, as you'll, as you get used to new systems, you'll find speed up again on them. So letting your delivery team know that product teams and the business that you're implementing a new way of working is quite is crucial to along for a successful adoption.

Training is also good.

The more you know, the more you can cater for things like, I think Jack mentioned it in his talk as well. Dev was launching our trail head as I named a few. Jack had a way more, Stephanie, keep that in, as well.

So yeah, based on that PR, you mentioned some have some of the new features that are coming? Yep. Yeah. So, before I mentioned, before we talk really quickly about specific teachers, just a few words about how we approach product development at Gayset. So we heavily heavily influenced by our customers and then what they need and expect from a modern DevOps platform we regularly talk to hundreds of teams to find out the most impactful products and features that we can build for them.

And we've got one, one thing we wanted to show you today, as we get. So Martin and I first connected at the end of last year. My mentioned that was they were experiencing kind of slower performance.

Also, I think Maty really wanted to make sure that, We can use source tracking, that Salesforce provides for monitoring changes. Hopefully, some of you are aware of that feature that you can switch on in production and the so the timing of that conversation was perfect because we were just doing research, with our customers to figure out how we can speed up the perform of comparisons, so that people can build their packages much faster.

So we jumped on a couple of calls.

We showed you some early prototypes, really excited to show you. It's a sneak peek into a feature that's going into a pilot soon. Hopefully, the demo will autoplay. Cool.

So this is actually Matt running it now. He's he's got a feature flag on his account. He selected an org as a source. And a dev branch as the target.

And we now have this slightly, new workflow here. And Mart is picking source tracking filtering option that's, that's actually connecting to source tracking API, pulling only just changes, so this is much, much faster. And those changes have been compared in place against the target. And you can see on the right different types, metadata types, kind of appearing in black, that means they compared.

So, yeah, this this is a new workflow we are working on. It's going to an open pilot, in the next few weeks.

So, yeah, watch out for any announcements, and you'll be to take part, and hopefully, your compas your comparisons will be much, much faster, this way. So I think Matt, you selecting custom field at this time and as usual, just going to the problem analysis and some checks before deploying it. So, yeah, super excited to show you that. It's, first public appearance.

And, just to mention, to find out more about the products in Future we're building. We've got a public roadmap. It's published on our website. We try we updated quarterly, trying to show you that kind of the next quarters and the things we are working on. So we've got a bunch of new cool stuff coming out. It's at Guesets dot comroadmap.

Yeah, that's it. And Thank you for coming to the