🚀 Fabric CI/CD: Automate Your Deployments with the Fabric CI/CD Deployment Tool

In this episode, I'm going to walk you through practically how we can use the new Fabric CI/CD deployment tool, which is a powerful open-source utility that brings DevOps-style automation to your Fabric workspace. This tool allows us to automate the deployment of Fabric items like notebook, data pipeline, event ops, environments, and so on. It supports promotion of artifacts from dev to test to production environment, and it integrates cleanly with Azure DevOps or GitHub for source control and can use a Python-based script to sync changes from Git repo to a target Fabric workspace. Let's get started. In order for you to follow along in this end-to-end data engineering project, you need to have three requirements satisfied. Number one, you need a Microsoft Fabric account. A trial account will be more than sufficient. Number two, you need to have VS Code editor with Python installed locally, and number three, you need an Azure DevOps account. So, in my case, I've got these three requirements satisfied. So, this is my Microsoft Fabric data platform, and for this demonstration, I have already provisioned two Fabric workspaces, dev and when I come to this tab, I can see the prod. Now the dev workspace will be used for building and testing new Fabric items and code, while the production will be used for deploying stable production-ready content for our end users. So, let's swipe to the dev.azure.com to create our project and repository. So, I'm going to use my Chrome browser, and this is dev.azure.com, and I can see my name. This is my organization, and I'm going to call this project Fabric, and I can give a description if I choose, and this is going to be private, of course, and I'm going to scroll to the bottom and click on create project. So the new project has been created, and in about 10 seconds or thereabouts, it should be fully up and running. There we go. Now I'm going to click on repos on the left-hand side, and then I'm going to click on that. So, for now, we do not have any files, so we need to initialize this repo in order to be able to integrate our code from Fabric. So, to do that, I'm going to scroll to the bottom, and then there's an option to initialize the main branch with a readme or gitignore file. So, I'm going to click on this initialize, and I'm going to wait for a couple of seconds, and this is now initialized with a readme.md file. Now this is the main branch. I'm going to click on this and create a dev that is development branch, which is going to be based on our main branch. So I'm going to call this one dev, and then I'm going to maintain this and click on create. Now we're going to be integrating our code, the Fabric items, to the dev, and we're going to be creating the pull request from the dev to the main branch, and then we proceed to the integration using VS Code editor. So, we have the dev, and I can come back to the app.powerbi.com, and I want to switch to the dev_ws. Now in this case, I'm going to create a notebook Fabric item, and I'm going to search for notebook and click on that. I'm going to give this notebook a nice name. So I'm going to call this one notebook for data processing, and once I'm done, I'm going to hit enter to commit. That's nice. Now I'm going to come to this code and get rid of this comment and paste this code. Basically, I just want to create a schema with all of these columns and then the sample data, and then we're creating a spark data frame, and then we just want to show the content of the data frame. Now I'm not going to run this code. I'm going to come to the dev_ws workspace. Now I'm going to click on workspace settings, and then I can click on the Git integration. Now we've got two providers. So yesterday I released a video on GitHub integration. So you can watch the video in the description box below. Now I want to focus on the Azure DevOps today. So click on that, and you can see my Azure Active Directory account. So I'm going to click on connect, and I'm going to choose my organization, which is Abiola007, exactly what we have at the top here, and then I want to choose my project, which is Fabric, and then the git repository is also Fabric. Now I'm going to choose the dev branch. Click on that, and as I mentioned yesterday, we can organize things by creating a git folder. So I'm going to call this one Fabric items, and once I'm done, I'm going to click on connect and sync. So this is going to start the process of creating and syncing the content in this dev_ws workspace. You can see your contents were synced, which is beautiful. Now I can return back to the DevOps repository, and I'm just going to right-click and choose the option to refresh, and there we go. So, we have our code deployed to the dev branch, which is cool. I'm going to expand, and I can see the notebook for processing, and I can see the metadata as well as the metadata and the code I pasted into the notebook, which is cool. Now because this is currently in our dev, we can see the option to create a pull request to the main branch. So, we want to basically, you know, match this to the main branch. So I'm going to call this one notebook creation, and I can choose to give a description, and once I'm done, I'm going to scroll down a little bit. I see the option to create a PR. So click on that. So we have no conflict. So I'm going to click on approve and then click on complete, and at the bottom, I'm going to click on complete merge. So we have this code now deployed to the main branch. So I can come to the main branch, and then I can see the code now promoted to this. So again, I can expand, I can see the notebook, and so on and so forth. Now I'm going to proceed to clone this repository locally using the VS Code editor. So to do that, I'm going to click on this clone option. I can see the HTTPS and then copy this to the clipboard. So I'm going to at the bottom launch my VS Code editor, and I can see this just a brand new one. So first, I'm going to see the option to clone a git repository. So I can click on this. Alternatively, I can even choose this and then see the clone repository. So click on that. Now you must see this clone from GitHub. So it doesn't matter whether it is GitHub or Azure DevOps; both work. So I'm going to paste the HTTPS and make sure I click enter on the keyboard. So this is going to ask me to choose a folder to clone locally. So, I'm going to, you know, clone it to my C drive, and I'm going to come create a folder, and I'm going to call this one Fabric CI/CD. Press enter, and I'm going to double click to go right into the folder and then select as repository destination. So, as soon as I click on that, I can see cloning git repository locally. So I'm going to wait for this to launch, and I can proceed. Now we have the option to open the cloned repository. Now you can open in a new window or time we can open in this existing VS Code window. Now I'm going to choose to open in the current window, and we can see the Fabric name here, and then when I expand, I can see the items, the notebook, and I can see the .platform, notebook-content.py, and again I can investigate all of this. I'm going to proceed to the terminal and install the Fabric CI/CD, which is going to give us all the packages and dependencies for this operation. So at the top, click on the terminal, and I'm going to create a new terminal, and I'm going to pip install Fabric-CD. Now if you're doing this for the first time, it's going to install all the packages and the dependencies for you. But for me, this requirement is already satisfied. Now I'm going to proceed to create a Python file that will contain a script to deploy our Fabric items from our version control repository to our target Fabric workspace, which is our prod_WS. So to do that, I'm going to create a new folder. So click on this file and then open folder. I'm going to step back a little bit. Now don't forget we have this Fabric CI/CD. So I'm going to go right in there, and I want to create this at the bottom and not inside this, you know, Fabric because this contains all our codes and items. So I'm...

notebook data pipeline event ops environments and so on. and it supports promotion of artifact from dev to test to production environment, and it integrates cleanly with Azure DevOps or GitHub for source control and can use a Python based script to sync changes from Git repo to

a target Fabric workspace. Let's get started. In order for you to follow along in this end-to-end data engineering project you need to have three requirements satisfied. Number one, you need Microsoft Fabric account. A trial account will be more than sufficient. Number two,

you need to have VS Code editor with Python installed locally, and number three you need Azure DevOps account. So, in my case I've got these three requirements satisfied. So, this is my Microsoft Fabric data platform and for this demonstration I have already provisioned two

Fabric workspaces dev and when I come to this tab, I can see the prod. Now the dev workspace will be used for building and testing new Fabric items and code while the production will be used for deploying stable production-ready content for our end users. So, let's swipe to the dev.azure.com to

create our project and repository. So, I'm going to use my Chrome browser and this is dev.azure.com and I can see my name. This is my organization and I'm going to call this project Fabric and I can give a description if I choose and this going to be private of course and I'm going to scroll to

the bottom and click on create project So the new project is been created and in about 10 seconds or there about it should be fully up and running There we go. Now I'm going to click on repos on

the left-hand side and then I'm going to click on that. So, for now we do not have any files, So we need to initialize this repo in order to be able to integrate our code from Fabric. So, to do that I'm going to scroll to the bottom and then there's an option to initialize the main

branch with a readme or gitignore file. So, I'm going to click on this initialize and I'm going to wait for some couple of seconds, and this is now initialized with a readme.md file. Now this is the main branch I'm going to click on this and create a def that is development branch which going to be

based on our main branch So I'm going to call this one def and then I'm going to maintain this and click on create. Now we're going to be integrating our code the Fabric items to the def and we're

going to be creating the pull request from the dev to the main branch and then we proceed to the integration using VS code editor. So, we have the dev and I can come back to the app.powerbi.com and I want to switch to the dev_ws. Now in this case I'm going to create a notebook Fabric item

and I'm going to search for notebook and click on that I'm going to give this notebook a nice name.

So I'm going to call this one notebook for data processing and once I'm done, I'm going to hit the enter to comment That's nice. Now I'm going to come to this code and get rid of this comment and

paste this code. Basically just want to create a schema with all of these columns and then the sample data and then we're creating a spark data frame and then we just want to show the content of the data frame Now I'm not going to run this code I'm going to come to the dev_ws workspace

Now I'm going to click on workspace settings and then I can click on the Git integration Now we've got two provider So yesterday I released a video on GitHub integration So you can watch the video in the description box below. Now I want to focus on the Azure DevOps today So click on that

and you can see my Azure Active Director account. So I'm going to click on connect and I'm going to choose my organization which is Abiola007 exactly what we have at the top here and then I want to choose my project which is Fabric and then the g repository is also Fabric Now I'm going to choose

the dev branch Click on that and as I mentioned yesterday we can organize things by creating a g folder So I'm going to call this one Fabric items and once I'm done I'm going to click on connect and sync So this going to start the process of creating and syncing the content in this dev_ws

workspace You can see your contents was sync which is beautiful Now I can return back to the DevOps repository and I'm just going to right click and choose the option to refresh and there we go. So, we have our code deployed to the dev branch, which is cool I'm going to expand, and I can

see the notebook for processing, and I can see the metadata as well as the metadata and the code I pasted into the notebook which is cool. Now because this is currently in our dev we can see the option to create a pulled request to the main branch. So, we want to basically you know match

this to the main branch So I'm going to call this one notebook creation and I can choose to give a description and once I'm done I'm going to scroll down a little bit I see the option to create a

PR So click on that So we have no conflict So I'm going to click on approve and then click on complete and at the bottom I'm going to click on complete match So we have this code now deployed

to the main branch So I can come to the main branch and then I can see the code now promoted to this So again I can expand I can see the notebook and so on and so forth Now I'm going to proceed

to clone this repository locally using the VS code editor So to do that I'm going to click on this clone option I can see the HTTPS and then copy this to the clipboard So I'm going to at the

bottom launch my VS Code editor and I can see this just a brand new one So first I'm going to see the option to clone a g repository So I can click on this Alternatively I can even choose this and

then see the clone repository So click on that Now you must see this clone from GitHub So it doesn't matter whether it is GitHub or Azure DevOps both work So I'm going to paste the HTTPS and make sure

I click enter on the keyboard So this going to ask me to choose a folder to clone locally. So, I'm going to you know clone it to my C drive and I'm going to come create a folder and I'm going

to call this one Fabric CI/CD. Press enter and I'm going to double click to go right into the folder and then select as repository destination So, as soon as I click on that I can see cloning git repository locally So I'm going to wait for this to launch, and I can proceed Now we have option

to open the clone repository. Now you can open in a new window or time we can open in this existing VS code window. Now I'm going to choose the open in the current window and we can see the Fabric

name here and then when I expand I can see the items the notebook and I can see the .platform, notebook-content.py and again I can investigate all of this I'm going to proceed to the terminal and install the Fabric CI/CD which going to give us all the packages and dependencies for this

operation. So at the top click on the terminal and I'm going to create a new terminal and I'm going to pip install Fabric- cd Now if you're doing for the first time it's going to install all the packages and the dependencies for you But for me this requirement is already satisfied

Now I'm going to proceed to create a Python file that will contain a script to deploy our Fabric items from our version control repository to our target Fabric workspace which is our prod_WS So to do that I'm going to create a new folder So click on this file and then open folder I'm going

to step back a little bit Now don't forget we have this Fabric CI/CD. So I'm going to go right in there and I want to create this at the bottom and not inside this you know Fabric because this contain all our codes and items So I'm going to come here and then right click

create a new folder and I'm going to call this one python script I'm going to press enter So I'm going to double click and I'm going to select the folder So this is going to automatically cd into that folder for me. Now I want to create a file and I'm going to call this one script

You can use whatever name you like and it must end with .py because this must be a python in code Now I'm going to come to my notepad and I'm going to copy all this configuration, So we have

this from Fabric CI/CD. So this we're going to be importing the Fabric workspace and the publish all item So we have this target_workspace equals to Fabric workspace class and then we have all of these parameters that includes the workspace id that is this going to be the target workspace and

then the repository directory which going to be our locally cloned repo directory and then these are the items that are supported which is item type in scope So for the workspace ID I'm going to come back here and then I want to go to the production that is the prod and then I can copy

the ID which is everything after this for slash to this point and I'm going to copy that Come back to the VS code editor and select this placeholder Ctrl + V to paste and then for the repository directory I'm going to come to my C drive and I'm going to go right into the Fabric CI/CD. Now, this

is the Fabric project So I'm going to go right in there and then I want to come to the back at the top and then copy this path to my C drive that is pointing to the Fabric items So copy back to

the clipboard Come back to the VS Code again I'm going to select this placeholder and Ctrl V Now we have to switch this backward slashes to forward slashes So I'm going to turn it to the for slash and then we're done So these are the items that are supported notebook data pipeline environment

event hubs and so on and so forth and then I've got this print authenticated as equals to target workspace and then we have this publish all items to the target workspace now this target is just a variable that is connecting to this production workspace id and then we are getting

all the items from this locally cloned environment so once this has been sorted I'm going to press ctrl s to save and then I can issue Now before I do anything let me quickly come to the product So you can see we do not have any items you know promoted to this environment So once this has been

cleared I can come back and then I can issue the run Python file and it's going to automatically launch the terminal and I'm going to wait for some couple of seconds and all things being equal we should be seeing some green lights telling us that this is working as expected So there we go So can

see publishing environment publishing notebook checking environment publish state and so on and so forth I'm going to swipe back to my web browser and I'm going to right click and choose the refresh So as soon as I refresh we're going to see the folder that we created and with the notebook

in it So there we go So we have the Fabric items folder Click on it Amazing We have the notebook for data processing. Absolutely fantastic! Now, I'm going to open this up We can see all the code So this is going to be our production environment Now I'm going to quickly come to the

the def and I'm going to quickly add more data to the bottom So just to show you something so I can understand you know how we can perform this CI/CD So I'm going to copy this Ctrl C and I'm going to

put in a comma Press enter Ctrl + V So I'm adding a new transaction at the bottom So when I go to the dev_ws I'm going to see the option that oh we have commit in the source control So I'm going to

click on this and I'm going to just create you know just call it created a new item and I'm going to select and then click on commit So we can see your selected changes were committed to the dev branch So I can go back to the the Azure DevOps and I can get rid of this I can go back to

the dev and as soon as I go to the dev I'm going to see the option to create a pull request Now before we do the PR. I'm going to come back here and expand the notebook and then when I scroll to

the very bottom I can see the record here So this is the duplicated record we just promoted Now I'm going to click on this create a pull request Again I'm going to maintain this name and I'm going to

go to the bottom create a pull request and then I'm going to approve and I'm going to click on complete So again click on complete match. So, again this going to be merged to the main branch in a matter of second Let's just wait for this to complete There we go So this is successful Now

when I come back to the VS code and write again in this script that is doing the deployment to the production workspace I'm going to again run the Python script So when I run this again and we go

to the production workspace you probably observe that nothing changes and I'm going to tell you the reason and what to do So we can see yeah this is done I can go back to the pro and switch here and

I'm going to go to the displays I can refresh just to make sure that everything probably synchronized and once that is confirmed I'm going to go to the notebook and when I scroll to the bottom I'm not going to see the new record So let me just go to the bottom a little bit So you can see I'm only

seeing the last record So the duplicate record is not found We need to go ahead and issue get pull origin main command in the terminal of our Fabric folder we cloned locally Now the command will fetch and merge the latest changes from the main branch into our local copy So how do we do

that practically I'm going to come back here and I'm going to switch back to that folder So again open this and then this is the folder that contains all our Fabric code and items So click on it and then I can select the folder again So you can see I just make the switch I just see the into

the environment So from this environment I want to go to the terminal Let me just wait for this terminal to get ready So this is now ready So I'm going to issue the git pull origin and I'm going

to explain So git pull origin and the origin that we want to point to is the main branch, okay. So now it is important you sign in You can see I'm properly signed into my GitHub So you need

to sign it in order for this for this to work. So get prior origin from the main branch So I'm going to press enter as soon as I press enter I'm going to wait and we can see remote Azure

repos and then we have this found six objects to send and we can see branch as the main So fetched head and then we can also see the Fabric items notebook for blah blah blah and then we can see

all of this So we've actually pulled the latest record or latest changes from the g into a copy of our local repository So once that is sorted I can swipe back to the Python script and I can go

ahead and run it again and once I run that we're going to see that we have the new record in the production workspace So I'm back here So I'm can you see I'm here So I'm going to click on

this again to run the Python script and I'm going to wait for some couple of seconds So we can see that in the terminal building up Okay So we can see validating you know publishing environment publishing data pipelines and so on and so forth So once this is fully sorted now I can go back and

check the production workspace I'm going to come back here So this is my you know prod and I can go right into this place again Uh let me just refresh just to synchronize the changes to the production workspace and I can expand the notebook for data processing and then when I scroll to the

bottom we should be able to see the record There we go Can you see So we have the record we just you know duplicated from the development branch which is cool Now what about if we create another five items How does it work So I'm going to come back to the dev workspace and I'm going to create

a pipeline just to demonstrate this So I'm going to call this one in a pipeline Just pipeline one is fine and there we go So once this is rendered in the workspace I'm going to see the option to

commit this to the dev environment So I can see the source control and I'm going to call this one I've created a new pipeline and I'm going to select the item and commit and there we go This is

going to be committed and I can come back to the Chrome browser I can swipe to the dev and then I can see the option to create a pull request So before I do that I'm going to expand this There we

go So we have the pipeline one and I can see the artifact as well as the content like maybe data set and so on and so forth So we want to create a pull request and the name is fine I'm going to

scroll to the bottom create a PR and then I want to approve and merge complete the merge and this is going to be merged to the main branch So once this is matched to the main branch I can

check it out by clicking on it and I can expand the Fabric item folder and there we go. This has been promoted to the main branch and again I'm going to come to this environment and I'm going to see the quickly to the Python folder and select this folder in order to be able to run the g pool

origin main because we need to actually pull this across to our local copy and then we can run the Python script on top So I'm going to say get pull origin and then this is going to be pulling from

the main branch and press enter. So when I press enter I'm going to wait for some couple of you in a moment and then we can see the changes So there we go So we have this pipeline and so on and so

forth We have the platform and then the content So once this is sorted I can come switch back to the Python script by clicking on this and then select the folder and go ahead and run this Python script again So click on this and let let's wait for that to run So this going to create another

instance for me of the terminal Okay Yeah So this now going So let's just wait and see the changes Okay So there we go Publishing workspace and so on and so forth So we have the pipeline going and this is fully deployed to the production workspace Again I can go back to the place click

on this Uh let me just come to the workspace itself Refresh and when I refresh I'm going to see the pipeline one deployed amazing to the production workspace So this is how we can use the Fabric CI/CD deployment tool to perform CI/CD using VS code Azure DevOps and

Fabric items So if you enjoyed this video give it a thumps up, share with friends, comments and subscribe to the channel because there is a lot to come Thank you for watching By now