Azure Event Hubs integration with Apache Spark now generally available

The Event Hubs team is happy to announce the general availability of our integration with Apache Spark. Now, Event Hubs users can use Spark to easily build end-to-end streaming applications. The Event Hubs connector for Spark supports Spark Core, Spark Streaming, and Structured Streaming for Spark 2.1, Spark 2.2, and Spark 2.3.

For users new to Spark, Spark Streaming and Structured Streaming are scalable, fault-tolerant stream processing engines. These processing engines allow users to process huge amounts of data using complex algorithms expressed with high-level functions like map, reduce, join, and window. This data can then be pushed to file systems, databases, or even back to Event Hubs.

Setting up a stream is easy, check it out:

import org.apache.spark.eventhubs._ import org.apache.spark.sql.SparkSession val eventHubsConf = EventHubsConf(“{EVENT HUB CONNECTION STRING FROM AZURE PORTAL}”) .setStartingPosition(EventPosition.fromEndOfStream) // Create a stream that reads data from the specified Event Hub. val spark = SparkSession.builder.appName(“SimpleStream”).getOrCreate() val eventHubStream = spark.readStream .format(“eventhubs”) .options(eventHubsConf.toMap) .load()

It’s as easy as that! Once your events are streaming into Spark, you can process them as you wish. Spark provides a variety of processing options, such as graph analysis and machine learning. Our documentation has more details on linking our connector with your



Unlock your data’s potential with Azure SQL Data Warehouse and Azure Databricks

Getting the most out of your data is critical for any business in a competitive environment. Businesses need the ability to get the right data into the right hands at the right time. Azure Databricks and Azure SQL Data Warehouse can help you do just that through a Modern Data Warehouse.

Azure SQL Data Warehouse is an elastic, globally available, cloud data warehouse that leverages Massively Parallel Processing (MPP) to quickly run complex queries across petabytes of data. Azure SQL Data Warehouse provides a familiar interface for your analysts who know SQL and want to drive action in your business.

Azure Databricks combines the best of Databricks and Azure to help customers accelerate innovation with one-click set up, streamlined workflows, and an interactive workspace that enables collaboration between data scientists, data engineers, and business analysts powered by Apache Spark.

With the general availability of the Azure Databricks Service comes built-in support for Azure SQL Data Warehouse. This enables any data scientist or data engineer to have a seamless experience connecting their Azure Databricks Cluster and their Azure SQL Data Warehouse when building advanced ETL (extract, transform, and load data) for Modern Data Warehouse Architectures or accessing relational data for Machine



Azure Databricks, industry-leading analytics platform powered by Apache Spark™

This blog post was co-authored by Ali Ghodsi, CEO, Databricks.

The confluence of cloud, data, and AI is driving unprecedented change. The ability to utilize data and turn it into breakthrough insights is foundational to innovation today. Our goal is to empower organizations to unleash the power of data and reimagine possibilities that will improve our world.

To enable this journey, we are excited to announce the general availability of Azure Databricks, a fast, easy, and collaborative Apache® Spark™-based analytics platform optimized for Azure.

Fast, easy, and collaborative

Over the past five years, Apache Spark has emerged as the open source standard for advanced analytics, machine learning, and AI on Big Data. With a massive community of over 1,000 contributors and rapid adoption by enterprises, we see Spark’s popularity continue to rise.

Azure Databricks is designed in collaboration with Databricks whose founders started the Spark research project at UC Berkeley, which later became Apache Spark. Our goal with Azure Databricks is to help customers accelerate innovation and simplify the process of building Big Data & AI solutions by combining the best of Databricks and Azure.

To meet this goal, we developed Azure Databricks with three design principles.

First, enhance



Announcing Terraform availability in the Azure Marketplace

In addition to Terraform already being integrated to the Azure Cloud Shell, I’m pleased to announce the availability of the new Terraform solution in the Azure Marketplace. This solution will enable teams to use shared identity, using Managed Service Identity (MSI), and shared state using Azure Storage. These features will allow you to use a consistent hosted instance of Terraform for DevOps Automation and production scenarios.

The Terraform solution configures Terraform to use Azure Storage instead of the local file system for Terraform state. This remote state implementation will lock state when one user is changing it, to allow multiple users to consistently change the state of shared environments, such as production.

The template also configures a Managed Service Identity and provides a Role Based Access Control (RBAC) script that will allow this identity to provision resources in the Azure subscription using Terraform. This eliminates the need for managing Service Principal secrets for Terraform separately in automation scenarios such as continuous deployment with Jenkins.

Azure Terraform Provider updates

Development on the Terraform Azure Provider also continues at a furious pace, we passed the 1.0 milestone last December, and version 1.3 has already shipped. As we near complete coverage of



Training State-of-the-Art Neural Networks in the Microsoft Azure Cloud

This is the third post in a three-part series by guest blogger, Adrian Rosebrock. Adrian writes at PyImageSearch.com about computer vision and deep learning using Python. He recently finished authoring a new book on deep learning for computer vision and image recognition.


In the final part in this series, I want to address a question I received from Mason, a PyImageSearch reader, soon after I published the first post in the series:


I noticed that you said tested all of the code for your new deep learning book on the Microsoft Data Science Virtual Machine (DSVM). Does that include the chapters on training networks on the ImageNet dataset as well? I work at a university and we’re allocating our budget for both physical hardware in the lab and cloud-based GPU instances. Could you share your experience training large networks on the DSVM? Thanks.

– Mason

Mason poses a great question – is it possible, or even advisable, to use cloud-based solutions such as the Microsoft DSVM to train state-of-the-art neural networks on large datasets?

Most deep learning practitioners are familiar with the “Hello, World” equivalents on the MNIST and CIFAR-10 datasets. These examples are excellent to get started as they require minimal (if any) investment in GPUs.



The next generation of Azure Alerts has arrived
The next generation of Azure Alerts has arrived

Today, we are announcing the general availability of the next generation of alerts in Azure. With Azure Monitoring Services, you can set up alerts to monitor the metrics and log data for the entire stack across your infrastructure, application, and Azure platform. With the release of the next generation alerts, we are providing a new consolidated alerts experience and offering a new alerts platform that will be faster and leveraged by other Azure services. Some of our customers have already been using the new alerts in preview and provided us feedback.

“The new unified experience dramatically improves our alert management capabilities. As part of our standard client configuration, we deploy and manage a variety of resource monitors to provide comprehensive coverage of a customer environment. These monitors include PaaS Resource Metrics/Logs, Azure Activity Events, and Log Analytics searches. We can now manage all of these monitors and alerts through a single interface and layer standardized action groups across all of them. This new service means we can offer a more consistent approach to our customers while dramatically reducing the management overhead. 

In addition, the performance improvements have allowed us to respond quicker to critical customer issues. With near real



Columnstore support in Standard tier Azure SQL Databases

We are pleased to announce the general availability of Clustered and NonClustered Columnstore indexes for Standard databases in the S3 and above pricing tiers. These features will enable a number of new and valuable scenarios:

Functional dev/test for columnstore based applications, without needing to pay for Premium tier databases for testing purposes. (Of course performance testing should always be done at the target performance configuration.) Deploying applications with columnstore-based storage which do not have the mission critical performance and availability requirements found in Premium tier database requirements. Application vendors can now develop an application which leverages columnstore functionality and deploy it on both Standard and Premium performance tiers. Columnstore advantages

Columnstore indexes are designed to be extremely efficient for queries which do scans and aggregations across millions and billions of rows of data. They are fundamentally different structures, which physically group data by column, rather than by row. In OLTP-style workloads, queries typically access one, or a few rows at a time, making traditional index structures the most efficient access path. For analytic queries, organizing data by column means that we only need to read the data for those columns involved in a query, and other columns never need be



Build your next iOS and Android game with $2,500+ of gaming services

To celebrate PlayFab’s first year at GDC as part of the Microsoft family, we are launching a special offer for developers featuring more than $2,500 worth of PlayFab, App Center, and Azure services free for up to a year.

This promotion is limited to the first 1,000 registrants and ends on May 31, 2018, so visit the promotion page now to claim the services.

What can I do with these free services?

Build, test, and monetize your next hit mobile game with a complete toolset across PlayFab, App Center, and Azure:


Use PlayFab services to build your game faster– then engage, retain, and monetize players through LiveOps tools.

Access all core PlayFab services up to 100,000 MAU, including player authentication, data storage, real-time analytics, optional data warehousing, segmentation tools, tournaments and leaderboards, remote config, triggered actions, and more. Build with the same services that power 80 million monthly active players across 1,200+ games. App Center

Use App Center to test your iOS and Android games on thousands of real devices in the cloud, distribute to beta testers, and monitor real-world usage with crash and analytics data.

Test on 3,300+ real iOS and Android devices in the cloud, with 440+



Coming soon to Power BI: Common Data Service for Analytics

https://powerbi.microsoft.com/en-us/blog/coming-soon-to-power-bi-common-data-service-for-analytics/Source: https://powerbi.microsoft.com/en-us/blog/coming-soon-to-power-bi-common-data-service-for-analytics/           New capability reduces the time, complexity and cost of extracting intelligence from disparate data sources.



SSMS 17.6 is now available: Managed Instance and many bug fixes

This post is authored by Alan Yu, Program Manager, SQL Server.

Download SSMS 17.6 and review the Release Notes to get started.

SSMS 17.6 provides support for almost all feature areas on SQL Server 2008 through the latest SQL Server 2017, which is now generally available.

In addition to enhancements and bug fixes, SSMS 17.6 comes with several exciting new features:

Added more support for Azure SQL Database Managed Instance. Fixed a key performance issue in SMO where scripting tables on SQL Server 2016 took 30 seconds, but now take less than one second. Object Explorer: Added settings to allow users not to force brackets around names when dragging and dropping from Object Explorer to Query Window. Data Classification: Improvements and bug fixes.

SSMS 17.6 also includes key bug fixes to Always On, SMO, and Database mail, which can be found in the Release Notes.

Azure SQL Database Managed Instance

Azure SQL Database Managed Instance (preview) is a new flavor of Azure SQL Database, providing near 100 percent compatibility with SQL Server on-premises, a native virtual network (VNet) implementation that addresses common security concerns, and a business model favorable for on-premises SQL Server customers.

Source from Managed Instance Documentation