Salesforce Continuous Integration and Continuous Deployment (CI-CD)

Document Version

Overview

This document contains information on how we will be managing our DevOps process for University of Sydney. This is applicable to all Salesforce deployments across multiple project streams.

The approach focuses on generalist engineering principals to make our solutions scalable, reliable and performant as possible.

There are multiple Salesforce implementations that are happening in parallel for University of Sydney. Currently, there are no guidelines on how to properly develop, deploy, and maintain the project’s package without making sure that the changes done for one project doesn’t overwrite the changes done for the other projects. Given that all projects use a single org, this is a common occurrence.

We will use Bitbucket Pipelines + SFDX to achieve this model.

Open

References:

Miro: https://miro.com/app/board/uXjVOxz-QN4=/

Presentation on CI/CD strategy: DEVOPS Strategy.pptx

The repository is found here: https://bitbucket.org/sydneyuni/uos-sfdx/src/master/

Scope

• DevOps will only be used on the following University of Sydney projects:

  • Hogwarts

  • Wellness

  • SingleCRM

  • Avengers

  • Other - if we want to add more projects to the pipeline, we can follow these steps: Adding new project to CICD pipelines

• For the initial release we will not be implementing destructiveChanges, but the implementation is scalable to do so. COMING SOON

Approach/Framework:

We use Bitbucket to maintain our repository. We are also using Bitbucket Pipelines for the CI/CD to have an easier link between Jira, Confluence, and the repository.

For our development,

• We use SFDX to pull and push changes between orgs.

• The team also agreed that since most of the team is not capable of using scratch orgs, we will be using sandbox orgs to represent the feature branches.

  • Having said this, the implementation is still scalable to accommodate using scratch orgs, because we are using SFDX. We can create scratch and then push whatever is on the force-app (script required to auto-install managed packages)

Limitation:

• The implementation relies on sfdx-git-delta, https://www.npmjs.com/package/sfdx-git-delta and if the differences on the org is not found through this plugin, we would need to

  • scale the process so that it detects that package.xml is custom COMING SOON

• This approach requires small knowledge on repositories and at least a minimum understanding of salesforce packages and xml

• We will be using the deployment from Hogwarts > SIT > UAT > PROD for the pilot release pilot release complete

  • Wellness and SingleCRM will be next in line on the upcoming versions. More soon COMPLETE

• Introducing new projects would require changes to the bitbucket-pipelines.yml

  • steps on how to add project can be found on this doc here Adding new project to CICD pipelines

  • this can be updated to make it become a boilerplate so no need to update every time a new project is introduced COMING SOON

Tools and processes:

Below are the tools required and the processes to follow to develop, test and deploy changes from one org to another

Prerequisites

• Bitbucket access - this is vital, because you cannot use CI/CD without this.

  • request license/access from IT

  • verify you have access by going to this link https://bitbucket.org/sydneyuni/uos-sfdx/src/master/

• VSCode https://code.visualstudio.com/download

  • Salesforce Extension pack https://developer.salesforce.com/tools/vscode

  • Git Graph (recommended, although optional) https://marketplace.visualstudio.com/items?itemName=mhutchie.git-graph

  • Optional - install sfdx-git-delta by typing this on the terminal sfdx plugins:install sfdx-git-delta

• Salesforce CLI https://developer.salesforce.com/tools/sfdxcli

• Git https://git-scm.com/downloads

• JRE https://docs.oracle.com/goldengate/1212/gg-winux/GDRAD/java.htm#BGBFJHAB

• Production access - Optional

  • this is needed ONLY when you want to use scratch orgs for your development. If you are not, skip this one

Initial set up

• Download and install all prerequisites

  • Install all the required apps mentioned on the Prerequisites section

• Authorise an Org

  • On VSCode, press Ctrl + Shift + P and find “SFDX: Authorize an Org”. Select it.

    • Open

      

    • Alternatively, you can click on the Status bar that says either your currently selected org, or if it’s your first time doing this, it will say “No Default Org Set”. after clicking you will see the same popup that let’s you Authorise an Org

  • select “Sandbox”

    • it’s usually a sandbox unless you want to create your Dev hub, in that case you select Production

  • set the alias. This will be the alias used for your credentials. It should represent the project, and the type of org it represents. eg:

    • uos-wellnessdev - I can understand straight away that this is a dev sandbox used by wellness project

    • uos-uat - I know just by looking that this is the UniSyd UAT environment

    • It is recommended to do so but you can name it whatever you like as long as you remember which one it is for

      • don’t worry, you can always use sfdx force:org:list if you forgot the alias

  • it should redirect you to a browser and will ask you to logon to the org.

    • enter your credentials and then click “Allow” if it prompts to ask if you are giving permission to Salesforce CLI

  • After logging in, go back to VS code and you should see the org alias on the status bar

• Clone the repository

  • go to https://bitbucket.org/sydneyuni/uos-sfdx/src/master/ and click clone, copy the URL and go back to VSCode

    • go to back to VS Code home page (if you’re not on the home page, press Ctrl + Shift + N) and click Clone Git repository…

    • it will open a popup asking for the Git URL. Paste the one you copied from https://bitbucket.org/sydneyuni/uos-sfdx/src/master/. Make sure you remove “git clone" from the copied URL. It should copy over the repository afterwards

  • Confirm that you have git connected by looking at the status bar. It should have the current branch you are on

• Create new branch using either of the following

  • via web https://bitbucket.org/sydneyuni/uos-sfdx/src/master/

  • command line git checkout -b <branch-name>

  • Git Graph - right click on the commit and select create branch

    •  

• Authorise Dev Hub - Optional

  • if you want to use scratch orgs, you can authorise a dev hub. This enables you to create scratch orgs. Otherwise, skip this step

• install required plugins

  • sfdx plugins:install sfdx-git-delta

Deployment/Build process

• Create a branch (follow the process mentioned above)

  • the most common place on where you create the branch from is the feature/* branch. This represents the central dev org of your project

    • e.g. feature/avengers represents all the contents inside the .avengers (Dev) org

• Name the branch similar to the Jira ticket. This is important because the automations on the pipeline would not work if you named your branch differently. For reference:

  • Hogwarts: HOG-xxxx

  • Single CRM: SINGLE-xxxx

  • Wellness: WCE-xxxx

  • Avengers: AVE-xxxx

  • Other - follow the steps here to define a project Adding new project to CICD pipelines

• Before starting development, you can pull the latest changes from org via:

  • sfdx:force:source:pull , although this cannot be used if the sandbox is created prior to us enabling source tracking in PROD

  • or sfdx force:source:retrieve although when using -p you won’t get the new ones created.

  • alternatively, you can use this script to retrieve from package dev-tools-scripts\oft-init-orgRefresh.sh -u <your-org-alias> -k manifest/package.xml , in which you can change the content of the package.xml if you like to exclude items or be specific on the retrieval

    • org alias is the alias of your authorised org in where you want to get the latest updates from

• Perform your build as per usual either from VSCode or directly on your sandbox

  • if your change is done directly on your sandbox, you have to perform another pull dev-tools-scripts\oft-init-orgRefresh.sh -u <your-org-alias> -k manifest/package.xml

    • Note: you can change the manifest/package.xml if you know specifically what you have changed.

• Commit and stage all changes. you will see all the changes here

• push to remote

• create pull request (PR) to feature/* (or to sit, uat, or master depending on what you are working on). In a PR, we can code review, comment on it, request changes, decline, or approve.

• Upon PR, as part of pipeline, it will automatically do the following:

  • compare the branch you created against the destination org, and then run all local tests

    • This does not deploy your changes, it just validates the package into the org

  • The package to validate is the delta of the changes, which means it will not include all metadata, it will only deploy the difference between the two branches.

  • You can check the printed package.xml to be deployed on the pipeline by clicking on the cat ./manifest/package/package.xml entry on the “Validating against destination org” step

  • if branch name starts with bugfix/ (branched from uat), it will automatically create 1 more pull request to sit IF the validation is successful

  • if branch name starts with hotfix/ (branched from master), it will automatically create 2 more pull requests to sit and uat IF the validation is successful

• once the pipeline is successful, approvers/requester can merge finally to destination.

  • pipeline will automatically get all the delta and deploy the changes into the destination after merging

• if the pipeline fails, you can check the deployment status on the org, you can correct them by updating the branch to fix the changes and the Pull request will automatically rerun the validation

Deploying specific features to Production

• If there are instances where only your project wanted to go to PROD, you would need to create a release/ branch

  • this ensures that only those items that you want to deploy (i.e. only the files that your team created) were deployed to PROD

• branch out of master and name it as release/xxxxx (replace xxxxx with an identifier e.g. release/SPM_Release2)

• update the .forceignore file with the files that you don’t want to include to your release. These are the files that are not part of the feature you wanted to release to PROD

• check out the newly created release/xxxxx branch and then merge uat into this branch

  • Most of the time, this will cause merge conflicts. Fix conflicts accordingly

  • you will still see the ignored files. That’s okay because when the package is validated, all the files you put on .forceignore will be ignored

• once conflicts are resolved, push this to remote and do a Pull Request between the release branch and the master

• You can check the package to be deployed listed as a manifest on the Pull Request by clicking the cat ./manifest/package/package.xml section

• You can also download the logs so you can copy the manifest and confirm the items that you are about to deploy before you commit

• approve and merge as per normal

Note: Few points to keep in mind for pushing your metadata to the feature branch are:

• Always commit the components related to the ticket only corresponding to the ticket branch.

• Always name the branch corresponding to the ticket number i.e AVE-**.

• Always put appropriate commit messages so that person reviewing the pull request knows what is this about.

• Check the pipeline if the pull request build is failed and if issues can be fixed.

• If there are many components, pull request should be small combining logical related components together.