Doria Hamelryk & Bogdan Ciureanu – Choose the right DevOps model for your project

Share with


Transcript

Okay. Hi, everyone. I think we're good to go.

Little bit of a intense session, but hope we won't lose anyone. It's both business focused, technical focused, high level, low level. You'll see what I'm talking about. But what this is helping you is to articulate where exactly are you in the the DevOps journey, level one to five. Maybe you'll imagine six in the future.

And this is why I said business focused because I I guess in your small, medium, large size program, you have a team lead, manager, especially a manager, and you will want to articulate where are you now, where are you around Christmas or Easter next and and we'll help with that. Okay.

So I'm Bogdan. I've already started. Doria will support me.

Let's go.

Okay. You hear me well? It's okay. Okay. All good. So for this, session, as we said, we will go through different levels, and the main goal is to talk about the maturity of each level because it's really important to understand where you are. Also, the ways to identify this level because sometimes it's not that easy to understand where we stand when we are working on the project.

And based on this, you can also define what are the business goals, what are the business benefits, and also the different steps needed to go from one level to the, to the next level. So we will try to give you some tools or at least some keys to identify your different levels and also the steps needed to evolve into your project.

And this is a journey we have been following since six years now. We are going in the sixth year, and I will be transparent later on. We're about step five with a with a step back in four, but still evolving.

Go. Okay. So let's go with the step one. It's, of course, illustrative. Each project has its own specificities, but just to give you an idea.

And I will talk for each step about what is concretely in place in the project. So So in the step one, it's basically the project just starting. You have a bunch of developer. You need to deliver something, and the main goal of the business is to go quick.

They don't care about the processes. They just go with the request. I want this. The developer develops, needs to be deployed.

The matter the the way it's deployed, the processes and everything we have to do best practices. Forget about this. We invested. We need business value quickly on the project, and this is what matter.

No long term projects neither.

I just go with the request as they come in on a daily basis. Of course, we have all the the changes, going on on the project because the organization is a bit messy because it's the first first stage because of this immaturity of, of the of the projects.

Processes very don't exist. The the workflow between between the team, let's forget about this. It's a kind of silo or mix between everything, everybody doing everything on the on the project. And when it comes to governance, of course, it's the same. Governance, not really existing yet because we don't focus on best practices nor guidelines.

So about that team structure where more of let's consider two developers, two islands, but still, generally a small team, two up to five, let's consider.

Still the drawback might exist that these two or three might override each other's work because they wouldn't know. It's no way to track. It's like in a small city, two cars three cars having an an accident. So it might happen.

I've seen it happening. Okay. And the risks are still piling up. It's without processes, without governance.

Yeah. Pipeline wise, what what real pipeline is like?

No no pipeline yet. No GitLab or other fancy things.

They these two, three developers have a shared mutualized so called sandbox, and still changes are being made in production, quite of a anti pattern. As I said, no version control, no processes, no real management. Yeah.

No pipeline. I mentioned this. I I would assume they would even use chainsets, which is a little bit of ancient history, maybe for some corner cases.

Yeah. What's on the slide is, ant migration script. This is something I picked up years ago. I used to work.

As far as I understood, it's it's abandoned. No further development offered by the community.

Dorian.

Yes. And, of course, when we take this first level, we could say, wow, this is not a level we should stay in, but don't forget also we have super tiny project. We don't only have big projects with a lot of budget, a lot of resources. And for those kind of project, it also has some benefits. For example, the cost. If you don't have the budget to, to set up a big team, well, you will stay in the this time and one, and you will be quite fine. Okay?

The because you will be able to start quickly, and also it's flexible because also not only on the technical side, but also on the business side, you will have some constraint. You don't have a PO. You don't have a BA. You don't have all this, whole organization to, to bring more clarity about the request.

So this also a kind of benefit of flexibility because you will be able to, to continue like this even if you don't have the team. Of course, as Bergen already mentioned, comes with the with the price, the errors. You will have a lot of errors because there will be things overwritten, and the the organization and the collaboration is not there. So you will face many issue in terms of, of deployment. This is what you really need to work on because if you want to increase the delivery speed of your of your project, decrease also the number of errors, and then, the, make the coordination better between the developers, then you need to switch to the next level.

Right. This is one of the mantras, but we have to deliver fast while keeping an eye on quality. So it's a interesting trade off. So huge time to market. We don't really want that unless it's one of these scenarios.

Okay.

So we started, and now we have a couple of weeks, months experience on the on the project. We are quite stabilized, and people, they know each other. We can start with the the stagnant stage of the of the implementation.

Correct?

Next. Sure.

Of course, the business goal. No. We started, so it's okay. And the business, they will want to to have more, like, say, security on what is delivered.

They don't want to have, each time, errors coming coming in. Of course, this will be common for all the stages. They want to to deliver fast. They want to live deliver properly.

But in the in the previous stage, it was not possible. So this time, we really focus on those kind of elements, the the reliability of the deployments, the quality, and also because, as we said, the the coordination between all all the members of the teams is really crucial. We want to bring also this kind of, a focus on the improvement.

For the project mat maturity, now we are at the early stage of, of the maturity, and we start, adopting some tools to improve the the deployment. We don't go for manu super manual things, and we introduce also the version control, bringing at least a centralized way of tracking the the changes.

And in terms of processes and workflow, it just starts to, to be defined between, by experience more than by strategy.

The this some guides are are built at this stage.

The governance is the same because we we had some kind of experience. We start to draft the the future governance, but we cannot say that the governance is really there. We just we are draft version of it. We're not we are not sticking to to the governance yet because, yeah, it's super light at this stage.

Regarding, team structure, this is getting more formalized.

Of course, the roles illustrated here are more functional, let's consider.

So can be one person covering multiple functions DevOps. Maybe he's a DevOps with some topics of release manager, and then we have a QA person and the the typical roles now.

With regards to the pipeline, we're already thinking and converting into SFDX. SFDX is not really a new thing now, popped up about five years ago, so not not yesterday news. But for, lots of programs, it was a cornerstone.

We cannot, accelerate without it. I I I can't imagine how. So it would also improve collaborations, in in regards to step two versus step one.

Version control is introduced, whatever it is, GitHub, GitLab, etcetera.

And, SFDX offers also command line tools. And it's a stepping stone for some best practices and you'll see in the step three for etcetera.

This is a snippet from, Visual Studio Code. Developers are generally free to pick up integrated development environment, whatever they they need. In our program, there are free free IDs already.

So next. Yep.

So the benefits first, the the velocity or the increase of, the frequency of the development. Now we have three two to four deployments per week. That's not that's something that we can reach, but at least it's, it's even better than having only once per week as we had in the previous stage.

The collaboration is also better because we we introduced some tools. We we also have less silos between the developer by having sharing tools, sharing sandbox, and so on. And, of course, there is more stability, into the deployment process, less, less errors.

Still, it's slow. It's slow because the the business wants to to have a fast value, also focus on innovation, not not really on the technical side of this. We we as tech people, we we focus on technique, but for the business, they they just see the the result of this. And they say, yeah, just two or four deployment per per weeks depending on the sector you're in. Sometimes it can be really, really limited.

Of course, we say that we introduce some tools, but it's not complete yet. So you we still have manual steps to, to do, and we who says manual step also say ever prone on this. And because of this, because it's not really fixed and secured yet, we have also a lack of trust or convenience in, in the quality of what we are deploying.

Yes. Let me just reiterate. And I'll quote one of the DevOps gurus. His name is, Jess Humble.

So Jez is saying, look, speed is a performance, software development performance indicator or predictor. If you don't go quite fast, faster, you will rarely really evolve into something better, get more corner cases cases. Yeah.

We'll in the wrap up, I'll come back to stress this point again.

So now we intro we we started we introduced some tools. We also introduced the collaboration. Now we are ready for the step three, which is build we call builder. K. And we will also work on other things.

Okay. So, as we said, velocity is everything. The business wants to to increase the the number of releases to four times per week. That was that was not enough.

And, of course, they also want to prepare for, scalability because it's not only to say, okay. We want to have four or six in case of two. No. And what if I want to increase to one per per hour, for example?

So we always need to keep the end in mind and say, okay. What about the scalability for the future?

The conflicts, still, still the quality issue to, to reduce and, of of course, goes with the speed and and processes.

The maturity is evolving at this, at this point. We have more workflow processes and collaboration across all of the team members.

The CICD is also implemented, and we will see it just just after this this slide. We also introduced other tools to enable more security, more reliability into the entire process.

Still, of course, we have, things to to fix or to improve. For example, the branching is, is one of, of, this point.

But at least in terms of governance, we are better. It's now it's more structured because, we are some couple of months later in, into this, this process. We are we are focusing on the the integration and the release cycle. We have a clearer, what is clearer governance and rules established for for the way all the people, they should work together and interact also together. And this, it was, yes, part of the just previous session about tests.

Test and CICD becomes also more important and really involved into all the rules and way of working of, of the company.

Still, it's flexible, the governance.

Team structure. So now we're kind of defined roles. It looks, much, clearer, crisp.

Release manager planning and overseeing, what what happens, as my colleague said previously, on the deploy area, including some quality gates. We have a QA engineer, fully dedicated DevOps engineer working on the pipeline, beginning automation tests.

So that clearly evolves.

I'm just reiterating, one developer, one dev sandbox. This this is like a better practice rather than stepping on each other's toes.

And with regards to testing environment, SIT with, running test test classes nightly and then UAT, we also made some large strides into securing UAT. Okay. Let's let's keep it as secure as possible, as close to prod. Let's not allow any cowboy on UAT. That's absolutely not normal.

On the DevOps pipeline, the the larger, improvement is considering Delta deployments. This is one of the one of the things that, at some point shifted old four branch deployments, which would take unimaginable time.

Consider we had, like, seventy k pieces of metadata in in our branch, so we couldn't deploy that much anymore. And now I'm sure many of of you guys know, SFDX get Delta, which is the bread and butter if you if you want to deploy, Delta. This is from, taken from their official site.

So we for, we don't need to deploy anything else except the the actual Delta. Yeah.

So as Doria said, releases are practically possible every hour or I would promise, and they have, like, on this, going from dev to to production, in in the same day because hotfixes might arise. And I I was asked how fast can you get the fix? Well, it's even faster than you develop the fix.

Could be if you spend half a day, then in the remainder in the afternoon, you'll get it on your testers in SIT to have a look at it, validate it, and deeper validating into UAT. In the in the evening, it will be in prod. What my colleagues said, I I didn't want to interrupt the guy before.

As we go up the stack C to A T, you build up some stability and confidence of stability that that stuff is quite good and will work on prod. I I don't really take an interest on SAT personally. I mean, I'm not downplaying.

But if it fails in SAT, then, okay, let's redo.

Let's go back.

But if it fails on production, fully different.

That's all I'm trying to say.

Yep.

So I would just say the main benefit, the the frequency daily release is quite good for as a global average for the for the timing.

Also, the reduced conflict because we have a better management in terms of branches and, also the tools used for this, and, of course, the scalability because now we have at least a basic tool to increase the number of, release we have predict.

Of course, this, this needs to be achieved by fine tuning the branching strategy. And also because till now, there is still a manual thinking, you know, all those situation when you have conflicts or, okay, back and forth between the the different environments. So this need to be fixed to go to a a real scalable, solution and go also more frequent than just once per day. So no refiner, fine tuning of, of the process.

No business, they focus on their business. They want to innovate. They want to innovate quickly. They, they they really don't want to, to be involved into all of this. They want to have quality. They want to be able to have the feature deployed really quickly and quasi near near time or near real time, let's say. And, of course, to, to maintain the speed, the quality, and the the security, in terms production with, with ten ten fixes coming just after that.

For the maturity, it's quite good already at step four. Honestly, if you already reached this, this level, I guess we are we are quite okay, because the DevOps process is already optimized. You have a team. They know how to, to interact. You have the tools in in place. Of course, it can be improved, but you you have all the all the basic. You also have autumn automated quality check for the for the entire the entire pipeline, and you also introduce other kind of tools that we will be talking about to increase, this, distrust.

The governance is there, and and now there is no point about, choosing to follow it, yes or no. It's, it's strongly implemented in into all, all the aspect of the organization.

It's also checked automatically by people, but also by, by automated mechanism and tools there. And we have, also enforced the quality level quality standards because at this point, once you have this pipeline built, you need to be sure that everything passing through this pipeline is also correct.

Team structure, I would just mention the fact what we're doing since three years is is like this.

We have about ten streams pushing various changes into SIT and then UAT. So it's like this one let's let's consider in SIT, it goes in the order of deployments one, two, three, four, five based on the branching strategy. It's like a modified git flow.

In UAT, it might go, four, five, one two three in or any permutation. So I'm sorry to say it it looks like, spaghetti a little bit, but so far it's stable. We didn't have, like, huge crashes ever in in three years. So it can go, I guess, twenty five permutations or or something.

And we are at that level, we have even a bigger structure than this.

The the core core structure actually are nine individuals with myself and also Doria.

So we're getting highly specialized doing day in and day out, things like this and optimizing and improving, every day. DevOps pipeline, it's more what is added on top is more stricter quality gates And, as also as per my former colleague just before this session, we are looking into PMD as a tool of choice, since since March this year. Sadly, only only since March. And, this is just a snippet of of how the rules these are some out of the box rules of PMD, and we are continuing to evolve every week on it with custom rules.

So we are for instance, I'll just give an example. We can catch various hard codings in what developers are pushing, and we will soon inform back, hey. You you pushed some hard coded value, hard coded email or name. You're not allowed.

After after Christmas, we will enforce, like, this pipeline cannot pass because, based on the severities of the rules, which we can manipulate in open source tool like PMD, we will say no go. Please please fix it. But, of course, in the in the DevOps, I'm just doing a parenthesis. In DevOps, adoption is a large part, so we will go slowly, like implement, inform, enforce, stop the pipeline, and then they they must fix it.

Okay.

Okay. Velocity at this stage, okay, super super fast. We We can go for multiple release, per per day if you if you want no no issue with that. Of course, the business is happy because now, the array of, of everything that is in place is super visible.

They can have a future, like I say, developing in the morning or in the in the afternoon. So no issue with that. And also security. The security becomes very, a crucial, point, at this stage, and we will talk about the the it's in the in the next phase.

I'll also give you some insight about the collaboration because DevOps is not only the risk management team. It's also a a kind of private team into an entire organization.

Of course, the because of this, it's more complex. So it means that you you have to to have a team specializing to this kind of also, you cannot have only journalists in your team and who say the specialized people means also budget. You need to, to be able to hire those, those people.

And the entire, I mean, the entire organization is more complex, so it's it's also something to, to keep in mind. We need also other people to manage those people.

Also, bear in mind, we came very long way step four. Just to reiterate, be mindful that we get to this point with lots of effort. We have a CICD pipeline that's clear.

Everybody needs to use it from top to bottom, top management to developer needs to be mindful.

We had a situation half a year ago with some people doing, some agile, but I I doubt it's agile because, development to production took twenty weeks.

So I I don't really appreciate that.

We, flagged this, so it goes quite slow. Okay. I I I get the scope is big, but let's try to break it down in sprints, like, three weeks. And now what rule of thumb we illustrated is please be ready at the end of the sprint to to deploy.

Still resistance, still backwards and forwards. It's like three three weeks sprint development, two weeks SIT, two weeks UAT, so that's seven, and then hopefully production deploy. So doesn't really work, but we're working on that.

Okay.

Last step or last stage, the leader.

So the leader, we really focus on scalability. Everything is in place. Now you need also to to improve the global organization and also maybe change the the the mindset about how you see the the release management team. So you of course, the the business, they they know really focus on the innovation. They, they they know that they can rely on the on the release management. They they can have the features ready.

They also know that silence there in release management.

Yeah.

Let's say, at least everything is in place. That that's okay. We also have less errors because the quality gates are there and also the processes, the governance, is there. So they really can focus on the on this, but still, what they want is to to be able to scale.

What if I I also acquire another company or I also go for another another cloud? This is what we also had recently. We added another cloud into into this deployment processes. So what if you you have to do this?

You don't have, six months to be ready for this. You have, like, one at maximum, and then and then you're done.

This is only achievable when you have a very high level of, of maturity, of course, and experienced people. You know how the business ought to work with the with the business, the synergies also, between the the teams. And as we said, it's a very broad, not only the people busy with the deployments.

And, of course, we have to keep in mind the the global, process, the improvement.

And once again, not only the technical side, the way we are working, but I just mentioned planning is a part of yeah. Okay. Planning is a bit part of the risk management, but it's, moreover, the a part of the the mentality and the organization of, of the team. So this is also something that you need to, to work on.

And, And, of course, for the governance, it's okay. Governance is there. We have, rules. You previously, it was like, please follow the rules.

Now it's if you don't follow, you don't pass. It's a complete block blocking stage that we, that we are implementing.

And maybe also to, to mention the security topic because I I think that we This is coming from top management Yeah. It's since one year with us.

Yep. And it's it's security and it's whatever business requirements want want. So new features, expanded features, new clouds. Okay.

But let's it's still security first. Security have shoved in repository scanning. So all of a sudden, everything in GitLab is already scanned. I've been asked to to to mimic with PMD as well what the repo scanning is doing, so that's also possible as long as I understand this other tool.

So it's really security being priority zero here.

Nobody wants exposure or attack surfaces.

So that's critical. Okay. Moving on, team structure. This is already cross functional, highly autonomous, and it's going fast. Also introduced the incident manager, security specialist.

Yeah. As of, this year, when there are perturbances or on production and it smells like bad security posture, everybody gets excited. So we have, weekly things. It it's normal to have, improved security posture.

Dedicated roles, that's that's clear.

DevOps pipeline, yeah, also fully integrated the feedback loops, on on the top with scratch orgs, with, seeding, if you want to go that way, if you're really going for it.

Various experiments are possible, security checks. So nowadays, one of one of the big, directions of where we're putting PMD is for various security checks.

For automation, this is, coming back to one of the mantras, no compromise on quality and security.

We always wanted to validate, meta and deploy very fast. This is not something I built, this snippet. It's from, the Internet.

And, let's go on.

No.

So just, have the the maximized value because of the speed, because of, the reliability of about what we, we were building.

The scalability, now we are able to add additional scope, pretty quick manner, and also the resilience, of course, the security and so on.

No. The drawbacks, it's even worse than than the oh, should I say the the previous stage because you you bring more complexity in the the global, organization and tools. And, of course, you also need to manage this with the expert once again to you have you need to have expert, and you need also people to to manage those, those experts.

Just maybe one one additional thing to say about this stage, and this is related to, what we talk about in terms of security. I'm more I'm more working the architecture side of Salesforce products and of enterprise level.

And what I saw is that at this stage, you really have a collection or merge of the activities between architecture and best practices and also the risk management side. For example, super low level of example, best practice, I want to have a description of of, on the field when the field is created or high level. I don't want to have a developer change in my connected app policy because it's just a developer, and this should be centralized. How do you do that?

How do you block that? It's impossible. You just need to, you see, to work hand in hand between the architect and the the risk management team to be able to, to put those kind of, of rules in place. And this this is where it becomes very interesting because you see, it's not any more about DevOps.

It's really broad and changing the mentality of everybody till the till the BA also.

Before wrapping up and just reiterating that what we have what we had here, it was really, investment in in talent, so us two plus about seven, over five years, but, the organization didn't truly invest in in tools like off the shelf, like Giset.

That would have been a different journey. So it's more of the the tools from the organization, which they had GitLab already.

They had talent and, open source.

So that's what we achieved over some years. And in order to wrap up okay. So this is a reiteration of, DORA, the DevOps research and assessment.

We were already discussing so as elite is multiple deploys per day, of course, careful on on quality, put everything everything possible in the pipeline. This is what I've seen across the industry.

Lead time for changes, very, very small. So, of course, we started from low, medium, high. This is just typical, matrix.

So we can recover fast. We can put changes in production if we deploy fast, but hopefully, we don't fail fast. So that's that's quite critical. We test it quite a lot, whatever it is. Even the pipeline is tested in in some UAT before any updates go higher, and we continuously protect production.

We can screw it up in some other environments, maybe, God forbid, but not never in production.

Assessment and next actions. So if this just goes into saying if you're at this step, you need to assess it.

What would it take to to become at next step from beginner to starter, builder, refiner, and leader?

Wait.

Wait.

K.

And I think we're done.

Questions?

So you you mentioned at the start that, you mentioned at the start that, potentially you were looking to continue that journey and move to stage six. What what are some of the things you'd expect to see at stage six?

I think deeper, security scanning effectively going into all kinds of features Git GitLab is offering, OWASP, and just to to to go and execute ninety nine hundred percent of the security, compliance, and constraints.

That's that has become, at least for us, in our context, the highest, most important stakeholder.

Yeah. So that's what we're trying to to execute p zero always.

So that's one of the directions.

And also the automation. And even even further automation. I think people, there there might be something after adoption where where everybody understand. Look, SIT due to some tips and tricks. So, and we might in one year, we might go and put this one level further of automation. Like, if you if you developers see that this is validated or good, you can click and deploy it yourself.

But I would just do that for SIT. So just just idea off the top of my head, what would be next steps. Maybe a slide in the in the next in the future, next slide with what we'd think as step six.

And ISVs are, of course, there to to help us.

So PDM work with inflows?

Sorry? Yes. To be honest, I don't think I've experimented with that. Okay. Yeah.

Yeah. I say okay. You say you you it will become maybe a super tricky custom rule if you want to I I got to find, for example, what what is the complexity of flow? How when do you decide?

Okay. Your flow is too complex. It doesn't pass the thing. Like, you have sixty elements in to it.

No. Sorry. Go back to, to Apex. Is this something that you could achieve?

Maybe a bit tricky, but definitely, yes. We introduced also not really, DevOps or release management, topics into the into the PMD stuff, like we said, security, but also best practices or gates of, yeah, security and security posture in terms of users, for example.

At the beginning, also, we had to visualize what is it, how many violations we had. So we we built up some matrixes and presented that. I think there was a question in the back.

What?

Yeah. Multiple clouds. No.

Yes.

Yes. Things like.

Yes. We have the core. So service cloud has a core. Then we also introduced data cloud, not to say the level of maturity of DevOps in when it comes to, to data cloud as an existent manual steps. But we also have, yes, separate orgs, for specific businesses. And maybe this will also be kind of a a six step.

Because for example, what about the the alignment? Because we don't talk a lot about safe and you see, scalability.

What about the alignment between all those products? How do we this was also one one challenge we faced recently because we are enterprise level, so it's not only about service service cloud or salesforce. It's about the entire technology landscape.

How do you ensure when you have ten different streams working on the same project, including different technologies, and they they you have the same deadline. You you only know about SIT deadline.

How do you ensure that you will be ready on time knowing that they all have their their own life cycle Yeah.

And release management processes?

There was a, like, a step back from the fact that, look, we introduced some other cloud, like Tata Cloud, but we didn't really align all of the teams with the product increment planning, PI planning, big room planning. So okay.

And we lost a lot.

Like, we had to replan everything and number of I think we're still working with ten ten teams plus complex architecture landscape. So, yeah, a little bit shameful that saying, guys, we need to postpone three weeks, shift to the right everything.

I think I have to answer your question. Yes. You will have technical issue because some product CPQ, you have meta and also data acting as meta kind of. We we say this in a lot of industry cloud. So you have the technical side, but you also for me, the most difficult part is the organization side or the political side into this.

Other questions?

On which side? CICD.

Well, it's, GitLab, and on top of it, lots of work and effort.

And and also injected the PMD via SFDX scanner engine equals PMD, kind of like this.

No. At this point.

No.

We're we're, for for backup and restore, you mean?

I I don't know.

You mentioned, like, a backup specialist for Yes.

Yes. This is being shaped up as we speak because we are in the process final stages to select the ISV for, backup and restore.

We're down to the final four. We have an idea who will be, like, runner-up winner, and this role is absolutely necessary, like, backup set. But, well, the business would like to have, like, a system administrator slash backup specialist to do to to run that, the backup maybe store.

And also it doesn't mean that because we mentioned it needs to be in your team. It can also be delegated to a nice, for example. Let me give you an example of the the usage of this.

This. You go you you see you evolve and then you change thing basic things like sell pick list and references and so on, and then you deploy. And after one week, oh, we have an issue. We need to roll back.

What do you do with your with all the data that were that were changed based on, on specific metadata that that we need to roll back? So it's super difficult to well, it can be managed, by manual scripts and, you see, data cleansing. But when you have the proper tool into internal data and metadata, you can align this. Not to mention also not all the tools, they are they are able to to do this kind of recovery, but this is where we also need to, to have the this tool because sometimes you you notice the the issue.

We had an issue, and they know the business noticed, like, one month later.

Okay.

Now we are stuck with that.

Yes, please.

Alright.

Like automated?

I don't think it's about tool rather than some some type of questionnaire.

Yeah. Might want to Google that. Should be yes.

You want to take that?

No. So sorry. I I didn't hear your your question.

Oh, yeah. That well, depends on the pro project, honestly. I I was on product two hundred people, and everybody was working with all the technicals too. You say this concept of, oh, and I mean, they they cannot use SFDX and so on.

There there were all this.

But, yes, I think that you need to also involve the the admins into, into this kind of Our admin is is involved.

So it was, like, I think, six to seven months prep for, SFDX conversion.

I personally built up, the resources list where every and the the maturity levels on what is SFDX were staggering was differentiating. Some people knew the hell out of it. Others, what exactly is it? I never heard of it. So we had to to build up training materials for all.

So we had also various people talking about this, online, like zero to quite very good in SFDX. So we put that, and effectively, the streams, these ten streams needed to allocate time in their sprint to ramp up with SFDX so we we could, convert it. And we had some other technical hurdles when we, over a weekend, converted to SFDX, but that's different story. We could do some slides on that. Other question?

Normally, in in DevOps, you have units of work that progress to the right, and that works perfectly fine and then you synchronize back.

Address that? How did you how could you stop that from happening?

Well, there are a couple of rules of of thumb. I always enforce, like, guys, if you want to to put, some type of abandoned or to be abandoned meta in in SIT, that might still be possible because it's the first place where meta is coming together, that's SIT, but you shall not put that in UAT unless you you you you're sure and you promise you want to put it in prod because otherwise, it creates all kinds of, inflexibilities and concerns.

We had, I think, last last year in the winter, we had this situation and it's absolutely chaos to to think that you can deploy something and months later unmerge it or no. And we're not going to discuss, like, rollback strategies of Salesforce. That's we didn't experiment on that.

So Okay. Okay. Basically. No. Don't do it.

Don't do it. Yeah. Try not to do it. It.

Yeah. Planning. But we can like that. We can also take the lunch to discuss further if you want.

Someone tell me soon.

Anything else?

Yes? You say you've been to what you're trying to find That's still in the future. Feature flagging, yes. Actually, we we used the the concept just to fend off something, some some other options which were on the table.

At some point, Doria knows the the business was even considering some cloud which we just have production out of it. So deploy, test, and just hope for the best in prod. Everything everything in prod. We just had prod. So it was it was quite interesting scenario and both of us started arguing against also saying that, look, if you want this, then the the vendor needs to think from day one to to start implementing feature flags. And then little by little, they dropped it.

Right? Yeah.

Alright. Okay. Thank you.

Thank you. Thanks.