We are excited to share the release of the Microsoft Azure HDInsight Metastore Migration Tool (HMMT), an open-source script that can be used for applying bulk edits to the Hive metastore.
The HDInsight Metastore Migration Tool is a low-latency, no-installation solution for challenges related to data migrations in Azure HDInsight. There are many reasons why a Hive data migration may need to take place. You may need to protect your data by enabling secure transfer on your Azure storage accounts. Perhaps you will be migrating your Hive tables from WASB to Azure Data Lake Storage (ADLS) Gen2 as part of your upgrade from HDInsight 3.6 to 4.0. Or you may have decided to organize the locations of your databases, tables, and user-defined functions (UDF) to follow a cohesive structure. With HMMT, these migration scenarios and many others no longer require manual intervention.
HMMT handles Hive metadata migration scenarios in a quick, safe, and controllable environment. This blog post is divided into three sections. First, the background to HMMT is outlined with respect to the Hive metastore and Hive storage patterns. The second section covers the design of HMMT and describes initial setup steps. Finally, some sample migrations are described and
Real-time location intelligence is critical for business operations. From getting real-time road data, to building asset-tracking solutions for navigating drone fleets. Today, we’re excited to highlight a customer, AirMap, whose software solutions rely on Azure Maps for real-time location intelligence in a new frontier of technology called dynamic mission planning for drones.
AirMap is the leading global airspace management platform for drones. AirMap’s Unmanned Traffic Management (UTM) platform enables the deployment and operations of safe, efficient, and advanced drone operations for enterprises and drone solution providers. Since 2017, AirMap has been part of the Microsoft Ventures portfolio and has chosen Microsoft Azure as its trusted cloud for its cloud-based UTM platform. AirMap offers open, standardized APIs and SDKs that make it easy for software developers to integrate AirMap’s intelligence services and capabilities into third party applications. This includes situational awareness of flight conditions, airspace advisories, and global airspace regulations. The AirMap developer platform also offers easy access to AirMap’s global network of airspace authorities, who offer notification, authorization, and more to drone operators on the AirMap UTM platform.
Figure 1: AirMap dynamically renders polygons representing different geographic areas subject to airspace regulations.
When faced with the decision of selecting
We recently introduced support for HDInsight in Microsoft Azure CLI as a public preview. With the addition of the new HDInsight command group, you can now utilize all of the features and benefits that come with the familiar cross-platform Azure CLI to manage your HDInsight clusters.
Key Features Cluster CRUD: Create, delete, list, resize, and show properties for your HDInsight clusters. Script actions: Execute script actions, list and delete persistent script actions, promote ad-hoc script executions to persistent script actions, and show the execution history of script actions on HDInsight clusters. Operations Management Suite (OMS): Enable, disable, and show the status of OMS/Log Analytics integration on HDInsight clusters. Applications: Create, delete, list, and show properties for applications on your HDInsight clusters. Core usage: View available core counts by region before deploying large clusters. Azure CLI benefits Cross platform: Use Azure CLI on Windows, macOS, Linux, or the Azure Cloud Shell in a browser to manage your HDInsight clusters with the same commands and syntax across platforms. Tab completion and interactive mode: Autocomplete command and parameter names as well as subscription-specific details like resource group names, cluster names, and storage account names. Don’t remember your 88-character storage account key off the top
We’re happy to introduce the new Grafana integration with Microsoft Azure Monitor logs. This integration is achieved through the new Log Analytics plugin, now available as part of the Azure Monitor data source.
The new plugin continues our promise to make Azure’s monitoring data available and easy to consume. Last year, in the v1 of this data source we exposed Azure Monitor metric data in Grafana. While you can natively consume all logs in Azure Monitor Log Analytics, our customers also requested to make logs available in Grafana. We have heard this request and partnered with Grafana to enable you to use OSS tools more on Azure.
The new plugin allows you to display any data available in Log Analytics, such as logs related to virtual machine performance, security, Azure Active Directory which has recently integrated with Log Analytics, and many other log types including custom logs.
How can I use it?
The new plugin requires Grafana version 5.3 or newer. After the initial data source configuration, you can start embedding Azure Monitor logs in your dashboards and panels easily, simply select the service Azure Log Analytics and your workspace, then provide a query. You can reuse any existing queries
The year 2018 was a banner year for Azure AI as over a million Azure developers, customers, and partners engaged in the conversation on digital transformation. The next generation of AI capabilities are now infused across Microsoft products and services including AI capabilities for Power BI.
Here are the top 10 Azure AI highlights from 2018, across AI Services, tools and frameworks, and infrastructure at a glance:
3. Microsoft is first to enable Cognitive Services in containers.
4. Cognitive Search and basketball
AI tools and frameworks
7. Open Neural Network Exchange (ONNX) runtime is now open source.
10. Project Brainwave, integrated with AML.
With many exciting developments, why are these moments the highlight? Read on, as this blog begins to explain the importance of these moments.
These services span pre-built
As 2018 comes to an end, I look at the technology landscape. I look at the kinds of hybrid scenarios our customers are developing. for example, we see Airbus transforming aerospace with Microsoft Azure Stack and I realize that this year has been amazing for developers that design, develop, and maintain cloud-based apps. Azure Stack has improved support for DevOps practices. You can use Kubernetes containers. You can use API Profiles with Azure Resource Manager and the code of your choice. You can review walkthroughs and tutorials on getting up and running with a development practice using a continuous integration pipeline. With Azure Stack, your apps can be developed in the cloud. You can code once and deploy to environments in Azure or in your local data center.
We are now seeing some of your favorite services from Azure arrive on Azure Stack. The Azure Stack team is also excited to come together with other members of the Azure Edge family, which include Data Box Edge, IoT Edge, and Azure Sphere. If you didn’t get a chance to attend Ignite 2018’s session on the Intellgent Edge check out the “Delivering Intelligent Edge and Microsoft Azure Stack and Data Box” session.
On December 18, 2018, the Azure PowerShell team released the first stable version of “Az,” a new cross-platform PowerShell module that will replace AzureRM. You can install this module by running “Install-Module Az” in an elevated PowerShell prompt.
Since January 2018, PowerShell has been a cross-platform product with the introduction of PowerShell Core. Therefore, it has also become a priority for Azure PowerShell to have cross-platform support. Because of the changes required to support running Azure PowerShell cross-platform, we decided to create a new module rather than make modifications to the existing AzureRM module. Moving forward, all new functionality will be added to the Az module, while AzureRM will only be updated with bug fixes.
Configure Az in your environment
Because both Az and AzureRM use the same dependencies with different versions, it is not possible to run Az and AzureRM side by side in the same PowerShell session. Thus, Az and AzureRM cmdlets cannot be used together in scripts and in interactive sessions. To ensure that a script does not try to import both Az and AzureRM modules in the same session, if you do not have many existing scripts that use AzureRM, we recommend that you remove
In this post, I’ll cover three free resources every developer needs for learning Azure. Dan Fernandez leads the team responsible for bringing our technical documentation and learning resources into a more modern experience that supports new capabilities that were impossible to deliver via MSDN. Recently, I invited Dan to record a few episodes of Azure Friday with Donovan Brown and spend some time showing off the work his team is doing to provide the best doc and learning experience.
1. Microsoft Docs
Last December, I wrote 4 tips for learning Azure in the new year, in which I included links to several resources, including the Azure documentation. In that post, I admit that I did a disservice by glossing over the revolution that Microsoft Docs truly represents – both internally and externally. Not only did it radically change how we create documentation, it improved how you can learn and use Azure.
Did you know that the Azure docs are not only open source, but it’s currently the fastest growing project on GitHub? In this episode, Dan shows off some cool features, a few tips & tricks, how you can contribute, and a
There is a new Azure PowerShell module that is built to harness the power of PowerShell Core and Cloud Shell and maintain compatibility with Windows PowerShell 5.1. Its name is “Az.” Az ensures that Windows PowerShell and PowerShell Core users can get the latest Azure tooling in every PowerShell on every platform. Az also simplifies and normalizes Azure PowerShell cmdlet and module names. Az ships in Azure Cloud Shell and is available from the PowerShell Gallery.
The Az module version 1.0 was released on December 18, 2018, and will be updated on a two-week cadence in 2019, starting with a January 15, 2019 release.
As with all Azure PowerShell modules, Az uses semantic versioning and implements a strict breaking change policy – all breaking changes require advance customer notice and can only occur during breaking change releases.
For complete details on the release, timeline, and compatibility features, check out the Az announcement page.
New features Az runs on Windows PowerShell 5.1 and PowerShell Core (cross-platform) Az is always up to date with the latest tooling for Azure services Az ships in Cloud Shell Az shortens and normalizes cmdlet names – all cmdlets use ‘Az’ as their noun prefix Az simplifies and normalizes
The Developer Economics Q4 2018 survey is here in its 16th edition to shed light on the future of the software industry. Every year more than 40,000 developers around the world participate in this survey, so this is a chance to be part of something big, voice your thoughts, and make your contribution to the developer community. This edition introduces questions about ethics, privacy, security, and project management methodologies in software development.
Is this survey for me?
The Developer Economics Q4 2018 survey is for all developers (professionals, hobbyists, and students) engaging in the following software development areas: web, mobile, desktop, backend services, IoT, AR/VR, machine learning and data science, and gaming.
What questions am I likely to be asked?
The survey asks questions related to developer skills, and experiences with dev tools, platforms, frameworks, resources, and more.
Your background and skills for demographics What’s going up and what’s going down in the software industry? Are you working on the projects you would like to work on? Where do you think development time should be invested? Which are your favorite tools and platforms?
Also, keep an eye out for some technology trivia interspersed in the survey. You may learn something