This blog post was co-authored by Ali Ghodsi, CEO, Databricks.
The confluence of cloud, data, and AI is driving unprecedented change. The ability to utilize data and turn it into breakthrough insights is foundational to innovation today. Our goal is to empower organizations to unleash the power of data and reimagine possibilities that will improve our world.
To enable this journey, we are excited to announce the general availability of Azure Databricks, a fast, easy, and collaborative Apache® Spark™-based analytics platform optimized for Azure.
Fast, easy, and collaborative
Over the past five years, Apache Spark has emerged as the open source standard for advanced analytics, machine learning, and AI on Big Data. With a massive community of over 1,000 contributors and rapid adoption by enterprises, we see Spark’s popularity continue to rise.
Azure Databricks is designed in collaboration with Databricks whose founders started the Spark research project at UC Berkeley, which later became Apache Spark. Our goal with Azure Databricks is to help customers accelerate innovation and simplify the process of building Big Data & AI solutions by combining the best of Databricks and Azure.
To meet this goal, we developed Azure Databricks with three design principles.
In addition to Terraform already being integrated to the Azure Cloud Shell, I’m pleased to announce the availability of the new Terraform solution in the Azure Marketplace. This solution will enable teams to use shared identity, using Managed Service Identity (MSI), and shared state using Azure Storage. These features will allow you to use a consistent hosted instance of Terraform for DevOps Automation and production scenarios.
The Terraform solution configures Terraform to use Azure Storage instead of the local file system for Terraform state. This remote state implementation will lock state when one user is changing it, to allow multiple users to consistently change the state of shared environments, such as production.
The template also configures a Managed Service Identity and provides a Role Based Access Control (RBAC) script that will allow this identity to provision resources in the Azure subscription using Terraform. This eliminates the need for managing Service Principal secrets for Terraform separately in automation scenarios such as continuous deployment with Jenkins.
Azure Terraform Provider updates
Development on the Terraform Azure Provider also continues at a furious pace, we passed the 1.0 milestone last December, and version 1.3 has already shipped. As we near complete coverage of
Today, we are announcing the general availability of the next generation of alerts in Azure. With Azure Monitoring Services, you can set up alerts to monitor the metrics and log data for the entire stack across your infrastructure, application, and Azure platform. With the release of the next generation alerts, we are providing a new consolidated alerts experience and offering a new alerts platform that will be faster and leveraged by other Azure services. Some of our customers have already been using the new alerts in preview and provided us feedback.
“The new unified experience dramatically improves our alert management capabilities. As part of our standard client configuration, we deploy and manage a variety of resource monitors to provide comprehensive coverage of a customer environment. These monitors include PaaS Resource Metrics/Logs, Azure Activity Events, and Log Analytics searches. We can now manage all of these monitors and alerts through a single interface and layer standardized action groups across all of them. This new service means we can offer a more consistent approach to our customers while dramatically reducing the management overhead.
In addition, the performance improvements have allowed us to respond quicker to critical customer issues. With near real
We are pleased to announce the general availability of Clustered and NonClustered Columnstore indexes for Standard databases in the S3 and above pricing tiers. These features will enable a number of new and valuable scenarios:
Functional dev/test for columnstore based applications, without needing to pay for Premium tier databases for testing purposes. (Of course performance testing should always be done at the target performance configuration.) Deploying applications with columnstore-based storage which do not have the mission critical performance and availability requirements found in Premium tier database requirements. Application vendors can now develop an application which leverages columnstore functionality and deploy it on both Standard and Premium performance tiers. Columnstore advantages
Columnstore indexes are designed to be extremely efficient for queries which do scans and aggregations across millions and billions of rows of data. They are fundamentally different structures, which physically group data by column, rather than by row. In OLTP-style workloads, queries typically access one, or a few rows at a time, making traditional index structures the most efficient access path. For analytic queries, organizing data by column means that we only need to read the data for those columns involved in a query, and other columns never need be
To celebrate PlayFab’s first year at GDC as part of the Microsoft family, we are launching a special offer for developers featuring more than $2,500 worth of PlayFab, App Center, and Azure services free for up to a year.
This promotion is limited to the first 1,000 registrants and ends on May 31, 2018, so visit the promotion page now to claim the services.
What can I do with these free services?
Build, test, and monetize your next hit mobile game with a complete toolset across PlayFab, App Center, and Azure:
Use PlayFab services to build your game faster– then engage, retain, and monetize players through LiveOps tools.
Access all core PlayFab services up to 100,000 MAU, including player authentication, data storage, real-time analytics, optional data warehousing, segmentation tools, tournaments and leaderboards, remote config, triggered actions, and more. Build with the same services that power 80 million monthly active players across 1,200+ games. App Center
Use App Center to test your iOS and Android games on thousands of real devices in the cloud, distribute to beta testers, and monitor real-world usage with crash and analytics data.
Test on 3,300+ real iOS and Android devices in the cloud, with 440+
Today, I am excited to announce the general availability (GA) of Azure Database for MySQL and Azure Database for PostgreSQL. The GA milestone means that, starting today, these services are bringing the community versions of MySQL and PostgreSQL with built-in high availability, a 99.99% availability SLA, elastic scaling for performance, and industry leading security and compliance to Azure.
Since we started the preview of MySQL and PostgreSQL on Azure in May 2017, we have accomplished a lot, increasing compute scale up to 32 vCores, offering a new Memory Optimized tier, ability to scale storage on-line independent of compute without impact to application performance, allowing greater flexibility in backup storage options, and achieving industry compliance with ISO, SOC, and HIPAA. We will be compliant with the General Data Protection Regulation (GDPR) when enforcement begins on May 25, 2018. In addition, with availability in 22 regions worldwide, these services are truly global. The reach of the services remains a key focus for us, and we continue to work on providing availability across all 40+ Azure regions, which we expect to deliver in coming months. In the video below, my colleague Sunil Kamath shares some of the key benefits of adopting Azure database
Gaining insights to customer traffic is imperative in providing a best in class user experience. With Traffic View, a new capability in Azure Traffic Manager, you can get rich information and powerful visualization that captures a heatmap of users’ experience. This allows you to understand traffic patterns globally and zoom in on specific geographic locations.
Improve performance by leveraging Traffic View
Today, we are happy to announce the general availability (GA) of Traffic View, announced in public preview last fall. Since then we have received enthusiastic adoption and feedback. Insights provided by Traffic View include geographic location of your user base, latency experienced from these locations, and volume of traffic. These insights can then be used to optimize placement of your workloads in various Azure regions. You can use the visualizations provided through the Azure portal or you can download the raw data to do custom analysis with Power BI or similar tools.
To provide finer granularity, Traffic View can now be enabled at a profile level rather than subscription level. Starting this week, as we rollout the changes, you will have to re-enable Traffic View for the profiles you need. Traffic View will be charged starting May 1,
We see strong customer interest in the Blockchain solutions that are available on the Azure Marketplace. As customers build production Blockchain scenarios, we understand that network reliability and health monitoring become increasingly important. Today, we are excited to announce significant enhancements to our Ethereum on Azure offering. These improvements are the first of a series of releases designed to enable production-ready consortium Blockchain networks suited for enterprise scenarios.
Through our customer engagements, we have identified and rolled out the following improvements:
High availability of Blockchain network presence: As a participant on a decentralized network, your nodes’ presence in the consensus process is critical towards achieving a fair and secure network. To ensure high-availability, we’ve incorporated cross-region Azure VM Scale Sets (Azure VMSS) and Azure Managed Disks. In the case that a node goes down, Azure VMSS will automatically recover the node, and join it back to the Ethereum network. As active nodes may change over time, we take care of maintaining an accurate boot-node list, allowing other members to join the network. Simplified deployment experience: Choosing the right network topology for your scenario can be difficult. We’ve invested significant effort into consolidating the various Ethereum network topologies into one experience, which allows
Today, we are excited to announce the support for backup of large disk VMs and set of improvements aimed at reducing the time taken for backup and restore. These set of improvements and large disk support is based on a new VM backup stack and are available for both managed and unmanaged disks. You can seamlessly upgrade to this new stack without any impact to your on-going backup jobs and there is no change to how you setup backup or restore.
This announcement combines multiple feature improvements:
Large disk support – Now you can backup VMs with disk sizes up to 4TB(4095GB), both managed and unmanaged. Instant recovery point – A recovery point is available as soon as the snapshot is done as part of the backup job. This eliminates the need to wait to trigger restore till data transfer phase of the backup is completed. This is particularly useful in scenarios where you want to apply a patch. Now you can go ahead with the patch once the snapshot phase is done and you can use the local snapshot to revert back if the patch goes bad. This is analogous to checkpoint solution offered by Hyper-V or VMware with
Spring fans, we are excited to announce full support of Spring Boot 2.0 on the Spring Boot Starters for Azure, already available on Spring Initializr. Plus, with new Java and Spring extensions for Visual Studio Code you can build production-ready apps and easily deploy them to the cloud.
Integrate Azure services with Spring Boot 2.0 apps
Spring Initializr enables developers to handle the dependency management and make the bootstrapping process for Spring projects much easier. Today, we’re announcing that the Azure Starters have been updated to support Spring Boot 2.0, so you can easily integrate cloud services with your application.
All of the Azure Spring Boot Starters are available on Spring Initializr:
Azure Support: Provides support for the Azure services below, plus all other services currently available via Spring Boot Starters. Azure Active Directory: Enterprise grade authentication using Azure Active Directory, integrated with Spring Security. Azure Key Vault: Manage application secrets using Azure Key Vault, providing Spring value annotation support. Azure Storage: Integration with Azure Storage, including object storage, queues, tables, and more.
You can create a Spring Boot 2.0 project on Spring Initializr, then search for “Azure” to find the Azure modules:
Spring and Java extensions with Visual