Author : All posts by ilikesql

21

Jun

Event trigger based data integration with Azure Data Factory

Event driven architecture (EDA) is a common data integration pattern that involves production, detection, consumption and reaction to events. Today, we are announcing the support for event based triggers in your Azure Data Factory (ADF) pipelines. A lot of data integration scenarios requires data factory customers to trigger pipelines based on events. A typical event could be file landing or getting deleted in your azure storage. Now you can simply create an event based trigger in your data factory pipeline.

As soon as the file arrives in your storage location and the corresponding blob is created, it will trigger and run your data factory pipeline. You can create an event based trigger on blob creation, blob deletion or both in your data factory pipelines.

With the “Blob path begins with” and “Blob path ends with” properties, you can tell us for which containers, folders, and blob names you wish to receive events. You can also use wide variety of patterns for both “Blob path begins with” and “Blob path ends with” properties. At least, one of these properties is required.

Examples:

Blob path begins with (/containername/) – Will receive events for any blob in the container. Blob

Share

20

Jun

The June release of SQL Operations Studio is now available

We are excited to announce the June release of SQL Operations Studio is now available.

Download SQL Operations Studio and review the Release Notes to get started.

SQL Operations Studio is a data management tool that enables you to work with SQL Server, Azure SQL DB and SQL DW from Windows, macOS and Linux. To learn more, visit our GitHub.

SQL Operations Studio was announced for Public Preview on November 15th at Connect(), and this June release is the seventh major update since the announcement. If you missed it, the May release announcement can be viewed here.

The June public preview release is focused on improving our Extensibility experience with the release of new extensions as well as addressing top GitHub issues.

Highlights for this build include the following.

SQL Server Profiler for SQL Operations Studio Preview extension initial release Azure SQL Data Warehouse extension Edit Data Filtering and Sorting SQL Server Agent for SQL Operations Studio Preview extension enhancements for Jobs and Job History views Build your own SQL Ops Studio extension Visual Studio Code Refresh Fix GitHub Issues

For complete updates, refer to the Release Notes.

SQL Server Profiler for SQL Operations Studio Preview

The SQL

Share

20

Jun

How to Do Distributed Deep Learning for Object Detection Using Horovod on Azure

This post is co-authored by Mary Wahl, Data Scientist, Xiaoyong Zhu, Program Manager, Siyu Yang, Software Development Engineer, and Wee Hyong Tok, Principal Data Scientist Manager, at Microsoft.

Object detection powers some of the most widely adopted computer vision applications, from people counting in crowd control to pedestrian detection used by self-driving cars. Training an object detection model can take up to weeks on a single GPU, a prohibitively long time for experimenting with hyperparameters and model architectures.

This blog will show how you can train an object detection model by distributing deep learning training to multiple GPUs. These GPUs can be on a single machine or several machines. You will learn how to perform distributed deep learning on Azure, and how you can do this using Horovod running on Azure Batch AI.

Object Detection

Object detection combines the task of classification with localization, outputting both a category and a set of coordinates representing the bounding box for each object that it detects in the image, as illustrated in Figure 1 below.

Figure 1. Different computer vision tasks (source)

Over the past few years, many exciting deep learning approaches for object detection have emerged. Models such as Faster R-CNN

Share

20

Jun

Location and Maps in Azure IoT Central powered by Azure Maps

Azure IoT Central brings the simplicity of SaaS for IoT with built-in support for IoT best practices and world class security and scalability with no cloud expertise required. We have been constantly adding features and true to the promise of SaaS applications, you can just start using new features right away to build production-grade applications without worrying about managing infrastructure.

This blog post is part of a series of blog posts you will start seeing for new features in Azure IoT Central in the upcoming weeks.

Azure IoT Central now leverages Azure Maps. A portfolio of geospatial functionalities natively integrated into Azure to enable users with fresh mapping data necessary to provide geographic context to their location aware IoT applications. We received several interests from public preview customers to leverage geospatial services for various use cases ranging from simply localizing their devices, validating location information, spatially referencing device locations on a map, to geofencing use cases around their devices. As any other property in Azure IoT Central, location metadata can be persisted on the cloud and updated either by the device itself (device properties) or the user (application properties). By integrating with Azure Maps, user can now give geographic context

Share

20

Jun

Location and Maps in Azure IoT Central powered by Azure Maps

Azure IoT Central brings the simplicity of SaaS for IoT with built-in support for IoT best practices and world class security and scalability with no cloud expertise required. We have been constantly adding features and true to the promise of SaaS applications, you can just start using new features right away to build production-grade applications without worrying about managing infrastructure.

This blog post is part of a series of blog posts you will start seeing for new features in Azure IoT Central in the upcoming weeks.

Azure IoT Central now leverages Azure Maps. A portfolio of geospatial functionalities natively integrated into Azure to enable users with fresh mapping data necessary to provide geographic context to their location aware IoT applications. We received several interests from public preview customers to leverage geospatial services for various use cases ranging from simply localizing their devices, validating location information, spatially referencing device locations on a map, to geofencing use cases around their devices. As any other property in Azure IoT Central, location metadata can be persisted on the cloud and updated either by the device itself (device properties) or the user (application properties). By integrating with Azure Maps, user can now give geographic context

Share

20

Jun

Microsoft deepens its commitment to Apache Hadoop and open source analytics

DATAWORKS SUMMIT, SAN JOSE, Calif., June 18, 2018 – Earlier today, the Microsoft Corporation deepened its commitment to the Apache Hadoop ecosystem and its partnership with Hortonworks that has brought the best of Apache Hadoop and the open source big data analytics to the Cloud. Since the start of the partnership nearly six years ago, hundreds of the largest enterprises have chosen to use Azure HDInsight and Hortonworks to run Hadoop, Spark and other Open Source analytics workloads on Azure. Also, during this time, Microsoft has become one of the leading committers to Apache projects, sharing its experience running one of largest data lakes on the planet, with the open source community.

Azure HDInsight

Azure HDInsight is a fully managed cluster service that enables customers to process and gain insights from massive amounts of data using Hadoop, Spark, Hive, HBase, Kafka, Storm and distributed R. Azure HDInsight offers the latest Hortonworks Data Platform (HDP) distribution and related Open Source projects on the Linux OS. The service is available in 26 public regions and Azure Government Clouds in the US and Germany.
 
The Big Data and Hadoop community has rapidly evolved over the past few years. Azure HDInsight has supported

Share

20

Jun

Microsoft deepens its commitment to Apache Hadoop and open source analytics

DATAWORKS SUMMIT, SAN JOSE, Calif., June 18, 2018 – Earlier today, the Microsoft Corporation deepened its commitment to the Apache Hadoop ecosystem and its partnership with Hortonworks that has brought the best of Apache Hadoop and the open source big data analytics to the Cloud. Since the start of the partnership nearly six years ago, hundreds of the largest enterprises have chosen to use Azure HDInsight and Hortonworks to run Hadoop, Spark and other Open Source analytics workloads on Azure. Also, during this time, Microsoft has become one of the leading committers to Apache projects, sharing its experience running one of largest data lakes on the planet, with the open source community.

Azure HDInsight

Azure HDInsight is a fully managed cluster service that enables customers to process and gain insights from massive amounts of data using Hadoop, Spark, Hive, HBase, Kafka, Storm and distributed R. Azure HDInsight offers the latest Hortonworks Data Platform (HDP) distribution and related Open Source projects on the Linux OS. The service is available in 26 public regions and Azure Government Clouds in the US and Germany.
 
The Big Data and Hadoop community has rapidly evolved over the past few years. Azure HDInsight has supported

Share

20

Jun

Traffic Analytics now generally available
Traffic Analytics now generally available

We are excited to announce the general availability (GA) of the Traffic Analytics, a SaaS solution that provides visibility into user and application traffic on your cloud networks.

Since the public preview, three months ago, the solution has analyzed several terabytes of Flow logs on a regular basis for network activity across virtual subnets, VNets, Azure data center regions and VPNs, and provided actionable insights that helped our customers:

Audit their networks and root out shadow-IT and non-compliant workloads. Optimize the placement of their workloads and improve the user experience for their end users. Detect security issues and improve application and data security. Reduce costs and right size their deployments by eliminating the issue of over-provisioning or under-utilization. Gain visibility into their public cloud networks spanning multiple Azure regions across numerous subscriptions.

This GA release includes enhancements that help you detect issues and secure/optimize your network, faster and more intuitively than before.

Some of the enhancements in this release are:

Your environment: Provides a view into your entire Azure network, identifies inactive regions, virtual networks, and subnets – for example, network locations with VMs and no network activity for further analysis. Detects malicious flows as they flow across application gateways,

Share

20

Jun

Traffic Analytics now generally available
Traffic Analytics now generally available

We are excited to announce the general availability (GA) of the Traffic Analytics, a SaaS solution that provides visibility into user and application traffic on your cloud networks.

Since the public preview, three months ago, the solution has analyzed several terabytes of Flow logs on a regular basis for network activity across virtual subnets, VNets, Azure data center regions and VPNs, and provided actionable insights that helped our customers:

Audit their networks and root out shadow-IT and non-compliant workloads. Optimize the placement of their workloads and improve the user experience for their end users. Detect security issues and improve application and data security. Reduce costs and right size their deployments by eliminating the issue of over-provisioning or under-utilization. Gain visibility into their public cloud networks spanning multiple Azure regions across numerous subscriptions.

This GA release includes enhancements that help you detect issues and secure/optimize your network, faster and more intuitively than before.

Some of the enhancements in this release are:

Your environment: Provides a view into your entire Azure network, identifies inactive regions, virtual networks, and subnets – for example, network locations with VMs and no network activity for further analysis. Detects malicious flows as they flow across application gateways,

Share

20

Jun

Azure Event Hubs is now offering support for Availability Zones in preview

Azure Event Hubs makes streaming data effortless because of its simplicity and ability to scale easily. The sheer volume of data that goes through the Event Hubs platform is a testament to how reliable the service is. In fact, by the time you finish reading this sentence, Event Hubs will have ingested over 100 million events globally.

Today, we are adding to the durability of the service and offering support for Availability Zones in public preview for Standard Event Hubs. This new feature adds even greater resiliency and fault tolerance to the top event streaming service. Support for Availability Zones partners nicely with our disaster recovery feature to offer a highly available service that can withstand both a zone outage and a regional one when both are properly utilized.

Event Hubs has customers in Retail, Auto, Finance, and other verticals that use its streaming capabilities for scenarios such as predictive analytics and financial trading. Availability Zones support further enhances our commitment to keeping our customers’ workloads running as smoothly as possible. We hope you try out this new feature.

What regions will this be offered? Central US East US 2 France Central

We’ll be offering support for additional Availability Zones

Share