Category Archives : Data Warehouse

20

Sep

Simplify modern data warehousing with Azure SQL Data Warehouse and Fivetran

Gaining insights rapidly from data is critical to being competitive in today’s business world. With a modern data warehouse, customers can bring together all their data at any scale into a single source of truth for use cases such as business intelligence and advanced analytics.

A key component of successful data warehousing is replicating data from diverse data sources into the canonical data warehousing database. Ensuring that data arrives in your data warehouse consistently and reliably is crucial for success. Data integration tools ensure that users can successfully connect to their critical data sources while moving data between source systems and their data warehouse in a timely yet reliable fashion.

Introducing Fivetran

We’re excited to announce that Fivetran has certified their zero maintenance, zero configuration, data pipelines product for Azure SQL Data Warehouse. Fivetran is a simple to use system that enables customers to load data from applications, files stores, databases, and more into Azure SQL Data Warehouse.

“Azure is our fastest-growing customer base now that we support SQL Data Warehouse as a destination for Fivetran users. We’re excited to be a part of the Microsoft ecosystem.”

– George Fraser, CEO and Co-Founder at Fivetran

We’re also pleased

23

Aug

Azure SQL Data Warehouse Gen2 now generally available in France and Australia

Today, we announce the broader regional availability of the industry-leading performance provided by Azure SQL Data Warehouse (Azure SQL DW). Azure SQL DW is a fast, flexible, and secure analytics platform offering you a SQL-based view across data. It is elastic, enabling you to provision a cloud data warehouse and scale to terabytes in minutes.

Compute Optimized Gen2 tier is now rolled out to three additional Azure regions— Australia Central, Australia Center 2, and France Central. These new locations bring the product worldwide availability count for Compute Optimized Gen2 tier to 22 regions.

Azure SQL DW Gen2 brings the best of Microsoft software and hardware innovations to dramatically improve query performance and concurrency. Our customers now get up to 5 times better performance, on average, for query workloads, 4 times more concurrency, and 5 times higher computing power compared to the previous generation. Azure SQL DW can also serve 128 concurrent queries from a single cluster, the maximum for any cloud data warehousing service.

Begin today and experience the speed, scale, elasticity, security, and ease of use of a cloud-based data warehouse for yourself. You can see this blog post for more info on the capabilities and features

21

Aug

Expanded Azure Blueprint for FFIEC compliant workloads
Expanded Azure Blueprint for FFIEC compliant workloads

We are pleased to announce general availability of the expanded Blueprint for Federal Financial Institution Examination Council (FFIEC) regulated workloads. As more financial services customers moving to the Azure cloud platform, we wanted to expand the Blueprint to explain how to deploy four different reference architectures in a secure and compliant way. It also takes the guesswork out of figuring out what security controls Microsoft implements on your behalf when you build on Azure, and how to implement the customer-responsible security controls.

We have built an end-to-end solution for moving compliant workloads to Azure, and reducing the time required to do so, leveraging Microsoft’s experience in working with banks in the U.S. and around the globe. Why start from scratch when you don’t have to? This Blueprint provides guidance for the deployment of PaaS Web Applications, IaaS Web Applications, Data Analytics, and Data Warehouse architecture in Azure suitable for the collection, storage, and retrieval of sensitive financial data regulated by the FFIEC.

The FFIEC Blueprint now consists of: Four reference architectures, with supporting deployment guidance Threat models to understand the points of potential risk Security control implementation mappings which describes how the reference architecture supports each control Customer responsibility matrix

02

Aug

Automatic intelligent insights to optimize performance with SQL Data Warehouse

We are excited to announce that SQL Data Warehouse (SQL DW) serves you intelligent performance insights within the Azure portal! SQL DW is a flexible, secure, and fully managed analytics platform for the enterprise optimized for running complex queries fast across petabytes of data.

Continuously delivering on fully managed experiences, customers no longer need to monitor their data warehouse to detect data skew and suboptimal table statistics with this release. Data skew and suboptimal table statistics are common issues that can degrade the performance of your data warehouse if left unchecked. At no additional cost, SQL DW surfaces intelligent insights for all Gen2 data warehouses and is tightly integrated with Azure Advisor to deliver you best practice recommendations. SQL DW analyzes your data warehouse collecting telemetry and surfaces recommendations based on your active workload. This analysis happens on a daily cadence where you can download recommendations, configure certain subscriptions to be analyzed, or postpone recommendations from being generated.

To check for recommendations, visit the Azure Advisor portal:

To generate these recommendations yourself, you can run the following T-SQL script and identify the specific tables being impacted by skew and statistics. For feedback on recommendations, please reach out

01

Aug

Microsoft Azure Data welcomes Data Platform Summit attendees

I am extremely honored to be delivering the keynote next week at the Data Platform Summit (DPS) in Bangalore. DPS is one of the events that I am looking forward to participating in as it will provide me with a pulse on data and analytics in the Asia Pacific region. More importantly, the event gives me a chance to connect with you — our customers, partners, and overall community.

As a preface, I want to share with you some of the exciting work happening on the Azure Data team at Microsoft and invite you to take a closer look. It is no secret that today’s mobile-first, cloud-first world is evolving to the intelligent cloud and intelligent edge, the new frontier for computing and data management. And IT decision makers and developers are at the forefront, driving this evolution.

We see three key trends that are defining and shaping the evolution of data management. 

First, data continues to grow exponentially and it’s not stopping any time soon. The explosion of data creates an urgency to deliver insights. Second, the rate of cloud adoption is increasing. Investments in public cloud technologies drive the need to collect data and innovate further at a

26

Jul

Avoid Big Data pitfalls with Azure HDInsight and these partner solutions

According to a Gartner 2017 prediction, “60 percent of big data projects will fail to go beyond piloting and experimentation, these projects will be abandoned”.

Whether you worked on an analytical project or are starting one, it is a challenge on any cloud. You need to juggle the intricacies of cloud provider services, open source frameworks and the apps in the ecosystem. Apache Hadoop & Spark are very vibrant open source ecosystems which have enabled enterprises to digitally transform their businesses using data. According to Matt Turck VC at FirstMark, it has been an exciting but complex year in the data world. “The data tech ecosystem has continued to fire on all cylinders.  If nothing else, data is probably even more front and center in 2018, in both business and personal conversations”.

However, with great power comes greater responsibility from the ecosystem. There is a lot more than just using open source or a managed platform to a successful project. You have to deal with:

The complexity of combining all the open source frameworks. Architecting a data lake to get insights for data engineers, data scientists and BI users. Meeting enterprise regulations such as security, access control, data sovereignty &

23

Jul

Accelerated and Flexible Restore Points with SQL Data Warehouse

We are thrilled to announce that SQL Data Warehouse (SQL DW) has released accelerated and flexible restore points for fast data recovery. SQL DW is a fully managed and secure analytics platform for the enterprise, optimized for running complex queries fast across petabytes of data.

The ability to quickly restore a data warehouse offers customers data protection from accidental corruption, deletion, and disaster recovery. We have seen scenarios where compliance requirements and having multiple test and development environments of a data warehouse enforce stricter capabilities in this area as well. To continue delivering first-class data protection and recovery, we have released the following critical improvements which are seamlessly integrated within the Azure Portal.

Finer granularity for cross region and server restores

You can now restore across regions and servers using any restore point instead of selecting geo redundant backups which are taken every 24 hours. Cross region and server restore is supported for both user-defined or automatic restore points enabling finer granularity for additional data protection. With more restore points available, you can be assured that your data warehouse will be logically consistent when restoring across regions.

Fast restore with Enhanced Restore Points

You can now restore your

23

Jul

Accelerated and Flexible Restore Points with SQL Data Warehouse

We are thrilled to announce that SQL Data Warehouse (SQL DW) has released accelerated and flexible restore points for fast data recovery. SQL DW is a fully managed and secure analytics platform for the enterprise, optimized for running complex queries fast across petabytes of data.

The ability to quickly restore a data warehouse offers customers data protection from accidental corruption, deletion, and disaster recovery. We have seen scenarios where compliance requirements and having multiple test and development environments of a data warehouse enforce stricter capabilities in this area as well. To continue delivering first-class data protection and recovery, we have released the following critical improvements which are seamlessly integrated within the Azure Portal.

Finer granularity for cross region and server restores

You can now restore across regions and servers using any restore point instead of selecting geo redundant backups which are taken every 24 hours. Cross region and server restore is supported for both user-defined or automatic restore points enabling finer granularity for additional data protection. With more restore points available, you can be assured that your data warehouse will be logically consistent when restoring across regions.

Fast restore with Enhanced Restore Points

You can now restore your

12

Jul

Lightning fast query performance with Azure SQL Data Warehouse

Azure SQL Data Warehouse is a fast, flexible and secure analytics platform for enterprises of all sizes. Today we announced significant query performance improvements for Azure SQL Data Warehouse (SQL DW) customers enabled through enhancements in the distributed query execution layer.

Analytics workload performance is determined by two major factors, I/O bandwidth to storage and repartitioning speed, also known as shuffle speed. In this previous blog post, we described how SQL DW caches relevant data to take advantage of NVMe based local storage. In this blog post, we will go under the hood of SQL DW, to see how the shuffling speed has improved.

Data movement is an operation where parts of the distributed tables are moved to different nodes during query execution. This operation is required where the data is not available on the target node, most commonly when the tables do not share the distribution key. The most common data movement operation is shuffle. During shuffle, for each input row, SQL DW computes a hash value using the join columns and then sends that row to the node that owns that hash value. Either one or both sides of join can participate in the shuffle. The diagram below

12

Jul

Azure sets new performance benchmarks with SQL Data Warehouse

As the amount of data grows exponentially, the pressure to quickly harness it for insights to share across the organization also increases rapidly. As Microsoft continues to evolve our analytics portfolio, we are committed to delivering a data warehouse solution that provides a fast, flexible, and secure analytics platform in the cloud.

Today we are excited to announce that Azure SQL Data Warehouse has set new performance benchmarks for cloud data warehousing by delivering at least 2x faster query performance compared to before. The key to this technical innovation is instant data movement, a capability that allows for extremely efficient movement between data warehouse compute nodes. At the heart of every distributed database system is the need to align two or more tables that are partitioned on a different key to produce a final or intermediate result set. Instant data movement in SQL Data Warehouse now accelerates this movement, resulting in faster query performance. You can learn more about how your query performance will improve from this blog.

Insights in minutes with Azure SQL Data Warehouse

We know that data makes every decision better, but decisions need to be timely to be competitive in the market. Fast decisions need