Author : All posts by ilikesql



Power BI Developer community April update
Power BI Developer community April update           This blog post covers the latest updates for Power BI Developers community- Power BI Embedded and Custom Visuals.




Adaptive caching powers Azure SQL Data Warehouse performance gains

Today we made Azure SQL Data Warehouse (SQL DW) Compute Optimized Gen2 Tier generally available to our customers. Even though data and data sources grow exponentially, organizations continue to demand faster and faster insights. Azure SQL DW Compute Optimized Gen2 tier delivers on this need with major performance improvements made possible through adaptive caching.

Analytics workload performance is typically determined by two major factors, I/O bandwidth to storage and repartitioning speed, also known as shuffle speed. This blog post looks under the hood of how Azure SQL DW exploits the latest hardware trends to improve effective I/O bandwidth available.

One of the recent hardware innovations becoming widely available are NVM Express (NVMe) solid-state drive (SSD) devices. NVMe SSDs offer significantly more I/O bandwidth than SATA SSDs or hard drives. A typical single NVMe device used in Azure, generally offers up to 2GB/sec of local I/O bandwidth, with multiple devices available per physical host, resulting in bandwidth previously reserved only to very high-end storage systems. Azure SQL DW Compute Optimized Gen2 tier fully takes advantage of NVMe devices through adaptive caching of recently used data on NVMe. With this breakthrough on customer workloads, we have observed up to five times the




Turbocharge cloud analytics with Azure SQL Data Warehouse

Data is transformative. The ability to turn data into breakthrough insights is foundational to remain relevant in an increasingly competitive market.

To help our customers deliver fast insights from exponentially growing data, today we are announcing the general availability of the Compute Optimized Gen2 tier of Azure SQL Data Warehouse. With this performance optimized tier, we are bringing the best of Microsoft software and hardware innovations to dramatically accelerate query performance and concurrency for our customers.

Fast, flexible, and secure cloud data warehouse

We launched Azure SQL Data Warehouse three years ago to make a powerful SQL based MPP (massively parallel processing) architecture data warehousing accessible to all. It was the first data warehouse that helped customers reduce costs by enabling them to scale compute and storage independently, and by offering pause and resume capabilities. This flexibility, combined with fast query performance, comprehensive data security, and governance capabilities has led to adoption by thousands of customers like Adobe, Toshiba, and LG Electronics. To keep up with customer demand, we have expanded the service to 33 Azure regions and it is now the most globally available of all cloud data warehouse services.

Azure SQL Data Warehouse Compute Optimized Gen2 tier




Region expansion for the next generation of SQL Data Warehouse

Azure SQL Data Warehouse (SQL DW) is a fast, flexible and secure, cloud data warehouse tuned for running complex queries fast and across petabytes of data. Continuing to deliver on this promise, we have announced the general availability of the next generation of SQL DW which includes an average of five times the performance boost, five times the increase in compute scalability, and four times the increase in concurrency. The release of Azure SQL DW Compute Optimized Gen2 tier comes with an expansion of 14 additional regions bringing the global region footprint of SQL DW Gen2 to 20 surpassing all other major cloud providers. The following regions are available:

Australia East

Australia Southeast

Canada Central

Central India

Central US

East Asia

East US

East US 2

Japan East

Japan West

Korea South

North Central US

North Europe

South Central US

South India

Southeast Asia

UK South

West Europe

West US

West US 2

With more global regions than any other




Blazing fast data warehousing with Azure SQL Data Warehouse

Today, we announced general availability of Azure SQL Data Warehouse (SQL DW) Compute Optimized Gen2 tier, the new generation of Azure SQL DW. Azure SQL DW is a fast, flexible, and secure cloud data warehouse tuned for running complex queries fast and across petabytes of data.

We see two key trends that drive data warehousing decisions, the amount of data continues to grow exponentially and the need to deliver insights from all this data is even more urgent. Azure SQL DW Compute Optimized Gen2 tier is designed to help customer accomplish just this by delivering dramatic query performance improvement. In addition, SQL DW now supports up to 128 concurrent queries while being able to provision five times more computing power compared to the previous product generation.

“After upgrading to the Gen2 of SQL Data Warehouse, our data warehouse workload has seen an average of 5.4 times performance improvement. This enhancement to the service is phenomenal and helps us deliver key customer insights for our business” said Brent Niezgocki, Senior Software Engineer for the Azure Active Directory analytics team at Microsoft.

Fast query performance through adaptive caching

As organizations look to accelerate time to insight, performance in the domain of




On-premises data gateway April update is now available
On-premises data gateway April update is now available           We are excited to announce that we have just released the April update for the On-premises data gateway. Here are some of the things that we would like to highlight with this month’s READ MORE




OS Disk Swap for Managed Virtual Machines now available

Today, we are excited to announce the availability of the OS Disk Swap capability for VMs using Managed Disks. Until now, this capability was only available for Unmanaged Disks.

With this capability, it becomes very easy to restore a previous backup of the OS Disk or swap out the OS Disk for VM troubleshooting without having to delete the VM. To leverage this capability, the VM needs to be in stop deallocated state. After the VM is stop deallocated, the resource ID of the existing Managed OS Disk can be replaced with the resource ID of the new Managed OS Disk. You will need to specify the name of the new disk to swap. Please note that you cannot switch the OS Type of the VM i.e. Switch an OS Disk with Linux for an OS Disk with Windows

Here are the instructions on how to leverage this capability:

Azure CLI

To read more about using Azure CLI, see Change the OS disk used by an Azure VM using the CLI.

For CLI, use the full resource ID of the new disk to the –osdisk parameter

NOTE: required Azure CLI version > 2.0.25

az vm update -g swaprg




Per disk metrics for Managed & Unmanaged Disks now in public preview

Today we’re sharing the public preview of per disk metrics for all Managed & Unmanaged Disks. This enables you to closely monitor and make the right disk selection to suit your application usage pattern. You can also use it to create alerts, diagnosis, and build automation.

Prior to this, you could get the aggregate metrics for all the disks attached to the virtual machine (VM), which provided limited insights into the performance characteristics of your application, especially if your workload is not evenly distributed across all attached disks. With this release, it is now very easy to drill down to a specific disk and figure out the performance characteristics of your workload.

Here are the new metrics that we’re enabling with today’s preview:

OS Disk Read Operations/Sec OS Disk Write Operations/Sec OS Disk Read Bytes/sec OS Disk Write Bytes/sec OS Disk QD Data Disk Read Operations/Sec Data Disk Write Operations/Sec Data Disk Read Bytes/sec Data Disk Write Bytes/sec Data Disk QD

The following GIF shows how easy it is to build a metric dashboard for a specific disk in the Azure portal.

Additionally, because of Azure Monitor integration with Grafana, it’s very easy to build a Grafana dashboard with these




Organizing subscriptions and resource groups within the Enterprise

Special thanks to Robert Venable, Principal Software Engineer in the Finance Engineering team of Core Services Engineering (formerly Microsoft IT) for sharing their story of enabling development teams while ensuring security and compliance. Thanks also to Scott Hoag, Principal Cloud Solutions Architect at Opsgility and Rob Dendtler, Account Technology Strategist at Microsoft for reviewing and providing invaluable feedback.

One of the common questions members of the Core Services Engineering and Operations teams frequently get when speaking to customers at the Executive Briefing Center here in Redmond is how do our engineering teams secure our Azure footprint for our Line of Business applications while still giving developers the freedom to go fast, have visibility into our environment and use the capabilities of Visual Studio Team Services for CI/CD, Release, and much more.

At the core of this answer is how we use the combination of subscriptions, resource groups, and Role Based Access Control to ensure compliance with a set of guidelines.

Let’s start at the top level: Azure Subscriptions. CSEO, as you can imagine has a lot of Line of Business applications, currently over a thousand. We loosely follow the business unit pattern from the Azure enterprise scaffold – prescriptive subscription




HDInsight tools for VS Code now supports argparse and Spark 2.2

We are happy to announce that HDInsight Tools for VSCode now supports argparse and accepts parameter based Pyspark Job submission. We also enabled the tools to support Spark 2.2 for PySpark author and job submission.

The argparse feature grants you great flexibility for your PySpark code author, test and job submission for both batch and interactive query. You can fully enjoy the advantage of PySpark argparse, and simply keep your configuration and your job-related arguments in the Json based configuration file.

The Spark 2.2 update allows you to benefit the new functionalities and to consume the new libraries and APIs from Spark 2.2 in VSCode. You can create, author and submit a Spark 2.2 PySpark job to Spark 2.2 cluster. With the backward compatibility of Spark 2.2, you can also submit your existing Spark 2.0 and Spark 2.1 PySpark scripts to a Spark 2.2 cluster.

Summary of key new features Argparse support – set up your arguments in Json format. Set up configurations: Go to command palate, choose command HDInsight: Set Configuration.

2. Set up the parameters in the xxx_hdi_settings.json file, including script to cluster, Livy configuration, Spark configuration, etc.

Spark 2.2 Support – Submit PySpark batch