About Gerhard Brueckl
Gerhard has been working with Microsoft BI tools since 2006 mainly focusing on Microsoft SQL Server and its components. As a consultant and architect he designed various enterprise BI solutions primary in the German-speaking areas. His personal interest has always been on analytical databases and their capabilities. From the start on he has always been active in the Analysis Services community and in 2012 he achieved the SSAS Maestro certification. This was also about the same time when he started his own blog and began speaking at international conferences all over the world.
7 months after the first official release of
Fabric Studio
, I am very happy to share the I just released the next major version with a lot of new features that make working with Microsoft Fabric from VSCode better and more intuitive than ever! The release includes some new capabilities that I wanted to get into the tool since the very beginning but back then the APIs just weren’t there yet. Finally they are and I integrated them into Fabric Studio v2.0!
The most important new features of v2.0 are listed below:
This has been on my list like forever as I had already implemented something similar for my
Power BI Studio
VSCode extension and its probably one of the features I use the most myself. It allows you to get a list of available API calls while you are typing in a
Fabric API Notebook
including examples with parameters from the official documentation.
While viewing the folders defined in the Fabric Web UI has been added to Fabric Studio already some time ago, there are some significant additions with v2.0. Most of them are about newly added Drag&Drop capabilities for Folders as well as Items. You can now drag an item or folder and drop it on a folder or a workspace and the item/folder will be moved to that folder/(root of the)workspace.
The PBIP format is becoming the standard when working with Power BI Desktop, especially in combination with Source Control and GIT. Fabric Studio now allows you to download any report or semantic model in PBIP format and seamlessly continue your development with Power BI Desktop. For reports you have the option to download the report with a live connection or including the semantic model metadata to be refreshed and worked with locally.
When working with Spark or data engineering in general in Microsoft Fabric, you will sooner or later come to the point where you need to reuse some of the code that you have already written in another notebook. Best practice is to put these code pieces into a central place from where it can be referenced and reused. This way you can make sure all notebooks always use the very same code and it is also easy to develop, update and test the common functions.
The first thing you would usually end up putting the common code in a dedicated notebook – lets call it
MyLibrary
and have a
%run MyLibrary
at the beginning of the referencing notebooks. This will basically execute the code from your
utils
notebook before the following cells. For simple scenarios this is usually fine but it can get quite messy when you start nesting those calls to
%run
and if the referenced notebooks get very long. Also, at some point you will realize that the performance suffers with the amount of code you import this way.
However, the nice thing about this approach is that you can develop your utils/library side-by-side with your notebooks and you do not need any external tools to manage your library which, especially for data engineers with little software engineering background, is a big plus as they do not need to learn any new tools.
Over time your library will grow and you will look for a “more professional” way to manage your library. That`s where python libraries, packaging and in the end Fabric
Environments
come into play. Now you are entering the world of software engineering which you may not be very familiar with. While this provides a lot of benefits, its also introduced a lot more complexity and a different way of working – usually outside of Fabric. A
.whl
file is built outside of fabric and uploaded and attached to an Environment in Fabric. Your notebooks then reference this Environment and inherit all settings from there instead of the Starter Pool.
This will significantly increase the development time for your regular notebooks, especially if you are still constantly changing or extending the library. Updating an existing environment just took about 15 minutes(!) + the increased cluster startup time. All in all not really great if your library is still in an early stage and changes frequently.
SparkContext to the rescue!
Fortunately, there is another way too which is not specific to Microsoft Fabric but a feature of Spark in general:
SparkContext.addPyFile()
This function allows you to inject custom Python/PySpark code into your current Spark context. The code file – a
.py
or
.zip
file – can reside anywhere as long as it is accessible from the Spark cluster. The typical file storage within Microsoft Fabric would be the
/Files
section of a lakehouse which can be used in any notebook.
Now here is what we want to achieve:
To accomplish this, I have created a generic notebook which I call
LibraryManager
. It allows you to define a set of notebooks which should be used and bundled into a library. The LibraryManager uses the Fabric REST API to download those notebooks as
.py
files into a lakehouse, compresses them into a
.zip
file and generates another notebook (
load_LibraryManager
) that then imports this
.zip
file using
sc.addPyFile()
. The notebooks that need to use your library can then simply use
%run load_LibraryManager
to import all the common code defined in your library. If the library evolves, the new functions will be available in all notebooks immediately without any further todos.
To make changes to the code in the library, I can now simply open the respective notebook in Fabric, and change the code as necessary. Then manually I run the LibraryManager notebook which only takes a couple of seconds to finish. Last but not least I restart the kernel of the referencing notebook and run it again which will load the most recent version of the library that I just updated. All-in-all this takes about 10 seconds and I do not need to leave Fabric.
Another nice aspect of this is that your library is now also part of your git repository and you always have a consistent state of notebooks and libraries and no dependency hell. The only thing you need to do is to run the LibraryManager notebook after each new deployment to make sure the library in the Lakehouse is up-to-date.
Now some people will argue that this is not professional and a library should have unit and integration tests and some proper versioning and release cycles and so on and so forth. And yes, you are right, for enterprise scale projects, that’s definitely the way to go! But if you only need a way to manage and share common code used in your data pipelines, this can still be a very good alternative with a low entry point and providing rapid development cycles and iterations not blocking you in any way and giving you a lot of flexibility.
Let me know what you think in the comments!
The
LibraryManager notebook
and a small sample of this can be found in my public github repository
Fabric.Toolbox
The workspace browser gives you an overview of all items that currently exist in your workspaces. This includes all items that currently exist and automatically extends to new items that might get added in the future. For selected items specific entries in the context menu were added e.g.
Copy SQL ConnectionString
,
Run Notebook
, …
There is also a common set of actions that exist for every item like opening the selected item directly in the Fabric Service via your browser or copy its ID or Name.
At the top you will find icons that allow you to filter the list of workspaces, refresh the current item, edit the items (e.g. semantic models, pipelines, … see below) or open a notebook that allows you to run arbitrary calls against the
Fabric REST API
.
Edit Fabric Items from VSCode
Using the context menu in the Workspace Browser you can select
Edit Items
which will open the definition of the selected item in your VSCode Solution Explorer as a new folder. You can either do this on the workspace level, a specific item type folder (Pipelines, Notebooks, …) or on an individual item. As of now, not all items are supported – here is a list of items that are supported as of now:
As Fabric Studio is solely based on the REST APIs provided by Fabric, I also wanted to offer a way to make running arbitrary API calls as easy as possible. The main problem when it comes to REST APIs is always authentication. As the API is already authenticated in the background, we can use the same mechanisms to also run any other API calls as well. Notebooks in VSCode offer an intuitive way to to do this. Another reason for this generic way of doing API calls is that not all endpoints will be covered by the UI so it just made sense to offer this option as well.
There would be a lot more features worth being mentioned here but instead I will create short demo videos and publish them via my social media channels (
Bluesky
,
X/Twitter
,
LinkedIn
). So to stay up-to-date with the most recent developments, make sure to also follow me there!
The last thing I want to mention is that the whole project is 100% open source and can be used under the
MIT license
. The repository is currently hosted in my GitHub account:
https://github.com/gbrueckl/FabricStudio
. If you are interested in the project and maybe want to contribute to it, please reach out to me!
If you like Fabric Studio but are working mainly with Power BI, make sure to also check out
Power BI Studio
– another extension developed by me, specifically tailored towards Power BI developers and admins!
