Calling a Fabric Data Pipeline from Azure Data Factory Pipeline
The Microsoft Fabric Data Pipeline is continuously evolving, with new features being added nearly every week. On the other hand, Azure Data Factory has been a stable data integration tool in the market for several years. If one wishes to trigger a data pipeline in Fabric from an existing Azure Data Factory pipeline, the Web activity within the Azure Data Factory pipeline can be utilized to initiate the Fabric Data Pipeline. This approach allows for the concurrent use of both data integration tools.
Given that Microsoft Fabric is a new service, the API is still evolving. The current API does not support the service principal authentication. We are using the username and password to generate the token. However, once service principal authentication becomes available, it should be utilized.
Here are the high-level steps.
Generating Token by using username and password. (not service principal)
(optional) please locate the Python scripts and Postman instructions for generating the user token in this repository. (https://github.com/Sam-Panda/FABRICation/tree/main/CICD/GeneratingToken/ )
This is a stop gap arrangement to generate the token by using the username and password. When Fabric API will support SPN authentication, we will switch to SPN authentication.
Prerequires:
- The user account that is used to generate token should not have the MFA enabled.
How to disable MFA for a user account?
There are various ways to disable the MFA, but the easiest way is to disable it from the entra.microsoft.com portal.
Identity -> Overview -> Properties -> Manage Conditional Access -> Multifactor authentication for Microsoft partners and vendors -> Users -> exclude the user from MFA.
2. Create a service principal for delegation and give API permission as it is required. I have given permission only for the workspace and item execution. If you would like to give more delegated permission, choose it from the Add a Permission -> search for Power BI Service -> delegated Permissions.
3. Also enable Allow Public client flows in the Authentication -> Advanced settings -> Enable the Allow public client flows.
We do not need the client secret for this script. So, do not create the client secret.
Creating the ADF Pipeline
We essentially have two web activities in the pipeline. The first web activity will create the user token, and the second web activity will trigger the Microsoft Fabric data pipeline using that token.
Pipeline Parameters
Here are the pipeline parameters:
web activity: Generate user token
URL: @concat(‘https://login.microsoftonline.com/',pipeline().parameters.tenant_id,'/oauth2/v2.0/token')
Method: Post
Body: @concat(‘username=’,pipeline().parameters.username,’&password=’,pipeline().parameters.password,’&scope=https://api.fabric.microsoft.com/.default&grant_type=password&client_id=',pipeline().parameters.client_id)
headers:
"Content-Type": "application/x-www-form-urlencoded"
Set token Variable task
name: access_token
value: @activity(‘Generate User Token’).output.access_token
Web Activity: Invoke Data Pipeline in Fabric
We have used the Fabric Data Pipeline REST API to invoke the data pipelines
From the Fabric data pipeline URL, we can get the pipeline ID that we want to invoke from Azure Data factory.
URL: @concat(‘https://api.fabric.microsoft.com/v1/workspaces/',pipeline().parameters.fabric_workspace_id,'/items/',pipeline().parameters.fabric_data_pipeline_id,'/jobs/instances?jobType=Pipeline')
Method: Post
Body: {}
headers: {‘Content-Type’: ‘application/json’,‘Authorization’: f’Bearer {access_token}’}
Hope this helps!
