In light of the recent COVID-19 pandemic - OPTASY would like to offer DRUPAL website support for any Health Care, Government, Education and Non-Profit Organization(s) with critical crisis communication websites or organizations directly providing relief. Stay Safe and Stay Well.

Drupal Migrate API: An Overview of the Migration System in Drupal 8

Drupal Migrate API: An Overview of the Migration System in Drupal 8

by Adriana Cacoveanu on Sep 30 2019

About to migrate your website from... Drupal 7 to Drupal 8? Or maybe from an external data source to Drupal 8? The good news is that Drupal Migrate API, the migration system in Drupal 8, is extremely powerful and conveniently flexible.

The "trick" is that you should be familiar with all its robust features, hidden gotchas and planning steps to take.

But, fear not: we've got you covered.

Here's a quick-start guide to migrating in Drupal 8, which covers both Drupal-to-Drupal and external data store-to Drupal migration scenarios.

 

1. But First: What Is Drupal Migrate API?

It's Drupal's robust migration system that enables you to pull data from various sources and to "inject" them into... Drupal (duh).

 

And the entire framework is made mostly of... migration plugins:

 

  • source plugins
  • process  plugins
  • destination plugins

     

It's these migration plugins that are responsible for extracting, transforming and fitting data into the Drupal 8 destination website.

2. A System Designed to Pull Content to Drupal from... Any Source

One of the Drupal Migrate API's most powerful features is extracting content from... pretty much anywhere:

 

  • XML
  • previous versions of Drupal
  • MariaDB, MySQL
  • JSON
  • CSV
  • other platforms (e.g. WordPress)

     

The overly simplified Drupal 8 migration process would look something like this:

 

  1. connect the Drupal Migration API system to your external datastore
  2. write your custom migration paths
  3. transform the source data into a suitable format to fit perfectly into the content types and fields in your destination Drupal website

     

3. Consider Creating Your Own Custom Source Plugin

Is there no way to pull content from your source? Then you'll need to write your custom source plugin to map the extracted data to its destination fields.

A plugin that suits your own migration scenario to the slightest detail.

For instance, mapping the titles and subtitles of the... blog posts on your current WordPress website to the article node type's titles and subtitle fields in your Drupal 8 destination website.

Drupal Migrate API- write a custom source plugin

IMAGE Image by Ulrike Mai from Pixabay  

You'll need to define the specific fields in your source data to the Drupal 8 Migrate API so that it should perform a proper mapping.

4. Process Plugins: The Key Elements in the Drupal 8 Migration Process

They play a critical role in any Drupal 8 migration, being responsible for converting the source data into the appropriate format. 

Here are just a few examples of process plugins "in action" during the migration process:

 

  • they import data behind HTTP authentication
  • they parse images from text

     

Drupal Migration API- process plugins

Image source: Drupal.org 

5. The Migration System Handles Everything... "The Drupal Way" 

The unparalleled flexibility of the Drupal Migrate API resides in its way of handling each operation... the Drupal way.

It's "smart" enough to automatically import and validate the source data into the right fields of the destination site. It's designed to "understand" how to adjust the imported data to the various fields, entities, and configurations specific to your Drupal 8 destination website.

In short: the Drupal 8 migrate system makes it easy for you to handle all migration operations the... "Drupal way".

 

6. Drupal 8 Migration Modules that Extend the System's Functionality

Now, let's say that, although extremely powerful, the Drupal Migrate API framework doesn't meet all your functionality requirements. You can always enhance its flexibility with migration modules that serve your particular migration needs.

Here are just 3 examples:

 

Migrate Plus

It adds extra features to the framework such as an URL source plugin that makes it possible for you to pull data from XML, Soap, and JSON file formats.

 

Migrate Tools

The module provides tools for various running and managing migration operations. Here are just some of the Drush commands triggering such operations:

 

  • migrate-import – it will run a migration.
  • migrate-status – it will display all migrations and their status
  • migrate-rollback – it will roll back a migration

Migrate Source CSV

It enables you to import CSV files to your Drupal 8 destination website.

 

Migrate Spreadsheet

It makes it possible to extract data from spreadsheet files...

 

7. External Data Source to-Drupal Migration: A 4-Step Guide

Now, let's talk facts.

Or, better said: the essential steps to take for migrating your content from an external source to Drupal 8.

 

  1. first, enable the Drupal 8 Migrate module
  2. next, install both Migrate Tools and Migrate Plus for the Drush migration commands and a whole variety of much-needed extensions and plugins
  3. set up a custom module for your specific migration case
  4. use YAML configuration files for field mapping from the right data source; trigger your process plugins to convert data to the right format

     

Note: your config files should get stored in “my_migration_module/config/install/“.

8. Drupal-to-Drupal 8 Migration: A 2-Step Guide

If it's a Drupal-to-Drupal migration challenge that you're facing, here's the whole process brought down to its key steps:

 

  1. run the "migrate-upgrade" command in Drush using the "-configure-only" flag to trigger stub YAML configurations    Upgrade Using Drush
  2. copy the resulting YAML files into the config/install directory of your custom module (remember to give them proper names and to edit them)

     

9. Final Word

The Drupal Migrate API is highly flexible and, therefore, very powerful. 

 

It's designed for a whole variety of migration scenarios, of different levels of complexity: from importing a collection of blog posts to... pulling hundreds of thousands of nodes.

 

And yet, with great flexibility comes great complexity:

Since every website's user-generated content architecture is different, every Drupal 8 migration is highly website-specific.

With no "one-size-fits-all" type of standards at hand, you're at the same time free and "constrained" to... customize your migration to your specific needs.

Better said: with great flexibility comes... great(er) responsibility.

Main image by wandaquinn from Pixabay  

Development

We do Web development

Go to our Web development page!

Visit page!

Recommended Stories

Drupal Performance Optimization: 17 Drupal Caching Best Practices To Speed Up Your Page Load Time- Part 2
"How can I make my Drupal 8 website faster?" Are you still struggling with this? Still striving to figure out which are the best (and most straightforward) Drupal performance optimization techniques for your website? Well, here I am today with a handful of 9 more ways that you can speed up your Drupal site. In addition to the 8 ones that I covered in the first part of this post. And yes: it's another round of Drupal caching best practices that'll help you boost your page load time. So, let's dive right into it: Tip #9: Use the Dynamic Page Cache Module  ... to cache for both authenticated and anonymous users. Unlike the Internal Page Cache module, that I mentioned in Part 1, which only caches pages for anonymous users. Tip #10: Use Distributed Cache, A Highly Effective Drupal Performance Optimization Technique But how does it work, more precisely? Once you've installed a distributed cache, it'll store your database's cache tables (Drupal's "cache_" tables) either in: file or memory Tip #11: Enable Drupal Cache for Anonymous Users Another one of those quick, yet powerful Drupal performance tuning steps that you can take. Tip #12: Use Squid to Cache Images and Static Content on Your Website "How to optimize Drupal for better performance?" You could go for Squid, an open-source caching proxy server. Now, since Drupal's already famed for its particularly dynamic content, the only cases where Squid does make a great performance booster are those where you need to cache static content. Tip #13: Add a Front-End Cache (i.e.Varnish Cache) Here's another handy Drupal performance optimization method for you: Use Varnish Cache to reduce the load on your server. How does it do it? It stores the HTML response, so that next time that the same page is requested, it serves it from memory. The result? Bypassed PHP and web server and... improved page load time. Tip #14: Use the Advanced CSS/JS Aggregation Module to Improve the Front-End Performance of Your Website  Combining your assets together is one of the most straightforward and effective ways to address those Drupal performance issues on your website. From: file grouping to caching to compressing ... the AdvAgg module handles all the steps that you need to take to aggregate your CSS and JS files. Tip #15: Install Memcache to Reduce Your Database Load You know how you're often struggling with keeping your database load to a minimum by caching database objects in RAM? In this respect, Memcache makes a great Drupal 8 performance optimization technique. It helps you reduce that load on the database and boost your page loading time. How? By taking standard caches out of the database. And by caching the results of resource-intensive database operations... Tip #16: Use the Entity Cache Module to Cache... Entities   Another caching best practice to boost Drupal 8 with is installing the Entity Cache module.  And its name says it all: it helps you cache entities. Tip #17: Cache Views  Here's the situation: Page requests made by registered users on your website lead to loads of queries to your database. Which impact the page load time. Now, to query the database, views are being used. And this is where this views caching module comes in handy to... boost things in there.   The END! These are our 17 recommendations for you on the best Drupal performance optimization methods for boosting your page load time. Not thrilled about the idea of having to go through the... Memcache installation process or to configure Varnish for Drupal? Or to put your current projects on hold so that your team can set up a... distributed cache? Maybe you don't have a professional Drupal maintenance team that could handle all these caching settings? We're here to help! Just drop us a line and let's figure out which of these 17 techniques are best suited for your website and the specific performance issues that it's struggling with. Let's speed things up in there! Image by Izwar Muis from Pixabay   ... Read more
Silviu Serdaru / Jun 23'2020
Drupal Performance Optimization: 17 Drupal Caching Best Practices To Speed Up Your Page Load Time- Part 1
"Why is my Drupal site so slow?" "How do I speed up my Drupal website performance?" In other words, what Drupal performance optimization techniques should you use? Which is the: most budget-friendly quickest most straightforward most effective ... solution to those Drupal performance issues that are slowing down your website? Caching... And luckily, Drupal 8 (it is a Drupal 8 website that you have, isn't it?) "spoils" you with one of the most advanced caching systems out there. The trick is that you follow the Drupal caching best practices and use it to its full potential. Speaking of which, here's a list of 17 such best practices: * I'll be covering 8 of them in this post, leaving the 9 remaining ones for the next blog post. But First: What Is Caching? "What is the purpose of caching?" "How does caching improve performance?"  2 legitimate questions that you might be yourself right now. Let me start by defining the Drupal caching process: Once a user accesses a page on your website, content elements and web data from that specific page (images, HTML, CSS, etc.) get stored in an accessible space. When that user visits the same web page again, your website will serve him/her the cached version of the content.  That if you haven't updated it since his/her last visit, of course... And this translates into: reduced bandwidth faster page loads Tip #1: Use the Internal Page Cache Module to Cache Pages for Anonymous Users   Say you have an "Add to cart" functionality for anonymous users on your eCommerce website. You can use this module to cache precisely this functionality. A Drupal performance optimization tweak that'll take you less than a minute to set up. Tip #2: Go for the Best Suited Tools for Heavy Traffic Drupal Sites Say you have a fairly busy Drupal 8 website. You've turned on caching in your performance settings, but... you haven't noticed any significant impact on your site's loading speed. So, you need to bring in the heavy artillery. To use powerful caching tools designed for high traffic websites. Here are some of the best tools and optimization techniques to try: switch to a Drupal-specialized hosting provider like Pantheon or Acquia move your database to its own VM/container (that if you still have it running locally, on your Drupal web server) upgrade to PHP 7.1.0 Enable OPcache via php.ini.  Put a proxy (i.e. Nginx) in front of your server Tip #3: Enable Block Cache - A Quick and Easy Drupal Performance Optimization Solution How to increase Drupal 8 performance? You cache those blocks that don't get updated frequently (like from one user to another). Tip #4: Use Views Content Cache to Update Upon Content Changes Only How does this Drupal module help you optimize your website for better performance? It allows you to expire views caches every time you update or remove content. The great thing about this caching method is that you get to cache blocks that appear on thousands of pages. Tip #5: Use a Content Delivery Network By far the most powerful Drupal performance optimization solution for your website. Why? Here are the 2 strongest reasons why you'd want to use a CDN to cache the static content (files, CSS, images, JS, fonts...) on your website: you keep the network delay to a minimum since your CDN has endpoints across the globe you get a better page loading time: your CDN has a domain different from your website's, so web browsers load content requests to your domain in parallel with the content coming from the CDN Tip #6: Set a Far Future Expiration Date for Your Static Assets Set up a "Newer expire" policy for your static components (e.g. use a far future Expires header) Tip #7: Use Redis as a Drupal Performance Optimization Technique to Store Large Amounts of Data Data that wouldn't fit into your server. "But what is Redis?" you ask? An in-memory store optimized for high-performance. Tip #8: Set the Maximum Time that Your Pages Can Remain Cached Another one of the Drupal caching best practices is setting the maximum amount of time that browsers should keep your cached data. The END of Part 1! And these are but 8 Drupal performance optimization solutions focused on caching. I have a whole list of 17 tips ready to share with you... So, stay tuned for another round of simple and effective caching techniques that'll help you speed up your website... But what if you don't have the time or the people in your team that you could assign tasks like: enable a block cache set up Redis  install the... views_content_cache module ...?  What if you could have a dedicated Drupal maintenance team implement all these performance optimization techniques on your website for you? We're ready to help you speed things up on your website. Drop us a line and let's set up the best caching strategy for your Drupal website. Image by mohamed Hassan from Pixabay   ... Read more
Silviu Serdaru / Jun 19'2020
What Are Some Good Examples of Drupal Sites for Nonprofit Organizations? Top 10 Nonprofit Websites Built with Drupal
Feeling a bit... uninspired? Or maybe you just don't see why you should consider Drupal for your NGO website? What if you could have a look at the top 10 nonprofit websites built with Drupal? And we're talking here about some of the world's most influential charity and non-profit organizations. Here are the 2 main criteria that we've used for putting together this top:   to be straightforward and easy to navigate to be visually pleasing and engaging: a clean and simple design helps the NGO's message shine through   Are these your own requirements, as well, for your non-profit website? Then, get ready to find plenty of inspiration by scanning through these great examples of what other non-profits have managed to do with Drupal: 1. Equal Opportunity Community Initiative  Who is EOCI?  The organization defines its own mission so clearly on its website: “... to build a world where children have equal opportunities to reach their dreams.” And all their projects and initiatives converge to meet this goal: providing equal access to educational resources for all children. Equal opportunity to compete and to succeed. The website that we've built for this international profit (revamping an age-old site, with little traffic and low conversion rate) is still one of the projects in our portfolio that we're most proud of. What makes it a great nonprofit website?   it's visually engaging: it tells a compelling story through visuals it's well-organized and easy to navigate: Drupal 8 has made it easier for us to structure content and to put together an information architecture that's easy to step through it's fast: the better the user experience, the higher the chances that they should turn themselves into the future volunteers or donors supporting the EOCI's initiatives   “The site’s beautiful and performing exactly how we’d hoped it would. The company that manages our Google Ads has found that our conversions have increased...  A couple of our donors are thrilled with the website and have committed their support again. They’re happy to be on the partners' page with a fresh look and approach to the public.” (Sean Kelly, Executive Director, Equal Opportunity Community Initiative, Source: Clutch.co)  2. Rotary International, One of the Top 10 Nonprofit Websites Built with Drupal  Who hasn't heard of Rotary International? A global network of community volunteers (1.2 million leaders and problem-solvers) actively involved in a wide variety of campaigns:   kicking Polo out of Africa providing clean water growing local economies supporting education   Why did they go for Drupal? For 2 strong reasons:   it's flexible: there's a module for almost any feature, any functionality they needed to implement to their nonprofit website it's extensible: no matter how "ambitious" the organization's future causes and campaigns and expectations from their website might get, Drupal's built to scale up to their growing needs 3. Doctors Without Borders Probably one of the best examples of nonprofit websites built with Drupal is Doctors Without Borders (Médecins Sans Frontières). A Nobel Prize-awarded organization of volunteering doctors, nurses, other staff, and journalists. Their mission? Providing medical aid to people affected by pandemics, epidemics, natural disasters, or armed conflicts, no matter their sex, nationality, religion, or political affiliation. Why Drupal?   because it's a multisite network that this global NGO needed and Drupal's famed for its built-in multi-site support. because Drupal's robust enough to withstand massive amounts of traffic (it's the biggest NGO in the world), all while being a conveniently flexible content management solution. 4. Habitat for Humanity International (HFHI)   Another one of the top 10 nonprofit websites built with Drupal. HFHI is an international, Christian NGO with a clear mission: building and renovating houses for low-income people.  To carry out its mission, it brings together donors, volunteers, and members of the families in need, who're co-opted in the process of building/repairing their own houses. How come their website is built on Drupal 8?   Drupal 8 is mobile-ready right out of the box it's open-source, which translates into lower costs it's flexible enough to empower their team to create, update, publish, and manage content on the go it scales to meet all the organization's future needs it ships with outstanding multi-language support it's fast 5. World Vision International "What are the most popular nonprofit websites that run in Drupal?" World Vision International's website is listed in any "top 5". No wonder:   the organization is the world's largest international children charity their website taps into Drupal 8's best features: easy third-party integration, freedom of customization, flexibility, and scalability guaranteed by its rich module collection   A nonprofit multisite that tells a series of highly compelling stories and grants a simplified donation process. 6. Human Rights Watch   A worldwide known independent organization, headquartered in New York, dedicated to protecting human rights.  And they define their own cause better than anyone else: "... we give voice to the oppressed and hold oppressors accountable for their crimes.”  Why did one of the most powerful NGOs in the world opt for Drupal?   because it's the best option for multilingual websites because it makes it easy to keep consistency while "joggling with" multiple content types  because it empowers content authors and editors to publish content quick and easy 7. Save the Children Spain   A member of the Save the Children International — the world's leading NGO dedicated to protecting and improving children's lives — Save the Children Spain has its website running on Drupal 8. And it is, by far, one of the 10 best nonprofit websites out there. Not just due to the global reputation of this humanitarian organization, but thanks to the complex needs that the Drupal website manages to meet:   it handles an entire network of operations that are critical for the NGO's activities: donations, news publishing, crowd fundings, training, collecting signatures, communicating with the press, etc. it withstands large influxes of traffic it supports a multi-site infrastructure it makes it easy for multiple teams (see Drupal's granular permission and access control system) to operate on the website, to integrate the tools they need for doing their work it integrates easily with all kinds of e-commerce third-party platforms (needed for their donation process, signature, membership, etc.) 8. Great Ormond Street Hospital Children's Charity   One of the UK's largest charities, Great Ormond Street Hospital Children's Charity (GOSH ) is dedicated to supporting the Bloomsbury-based children hospital and its patients. How? By focusing their efforts in 4 major areas:   supporting the little patients and their families investing in rebuilding and renovation projects pioneering research into children's health investing in life-saving medical equipment   When did Drupal come into the picture?  When the charity looked for a platform that would provide them with:   scalable architecture for their website, robust enough to scale up and down, to their fluctuating levels of traffic multi-site support, out of the box: GOSH.org is made out of 2 large websites and an ecosystem of 10 different sub-sites easy third-party integration: they needed to keep the same staff, so they looked to streamline their team's work, by integrating time-saving tools  9. The Wildlife Trusts We couldn't have left the Wildlife Trusts website out of our list Drupal nonprofit organization websites that stand out. The UK-wide group, comprising 45 local Wildlife Trusts, shares a common interest in preserving the wildlife and the biodiversity in their local area. And we're here talking about 2,300 nature reserves, on 98,000 hectares. 46 different independent charities, each with its specific activities, constitution, and membership criteria, under one “umbrella”? This multi-site project had “Drupal” written all over it... Now, let me outline, briefly, the key reasons why Wildlife Trusts has chosen Drupal for its website:   it provides great multi-site capabilities: the organization wanted to empower each trust to control its own content creation and publishing process, but they still needed some sort of a “central hub” it provides outstanding content management capabilities, much needed in this case, where huge volumes of content had to be managed effectively it meets their need of delivering the best mobile user experience it empowers the Wildlife Trusts editors to tailor page layouts as needed: and we're talking here about a multi-site that's rich in stunning visuals and video content 10. Top 10 Nonprofit Websites Built with Drupal: Allard Prize for International Integrity Another project from our portfolio that we're particularly proud of. The Allard Prize for International Integritiy is a photography competition that recognizes and awards people and organisations standing out through their efforts in fighting corruption and defending human rights. Why did they go with Drupal?   because they wanted to give their website a performance boost because Drupal provides them with a reliable translation system, a crucial feature when dealing with a global audience In Short, Why Would You Choose Drupal for Your Nonprofit Website? Here's a short inventory of the main reasons why you'd choose Drupal for setting up your nonprofit website:   you save money (Drupal's open-source) you get your multi-site, multi-language website ready to go in no time (especially if you opt for a Drupal distribution) you're free to further extend your website's capabilities by adding new modules and integrating new third-party apps  you get robust SEO features right out of the box you get a website that scales to fluctuating levels of traffic  you get to categorize, neatly structure, create, edit, publish, and manage your content quick and easy you're free to customize your page layouts to your liking   Need help setting everything up? Just send us a message with your feature requirements and specific expectations from your future nonprofit website and let's get your message out there. ... Read more
Adriana Cacoveanu / Jun 05'2020