Security is very hard to bolt on to any software or product after it has been built. Building it into the core of the code helps to avoid mistakes, and thus the upcoming release of Drupal 8 tries to build in more security by default, while still being usable for developers and site builders. This list of 10 security improvements is not exhaustive - some are just a line or two to handle an edge case, and there are others I may have overlooked. I've contributed to a number of these improvements, but they reflect overall the community consensus as well as reactions to problems that required security releases for Drupal core or contributed modules in the past. For each point I've tried to include a link or two, such as the Drupal core change record, a documentation page, or a presentation that provides more information. Some of these may also be possible to back-port to Drupal 7, to benefit you even sooner. A "7.x back-port" link indicates that. For context on why these 10 improvements are important, I looked at past security advisories (SAs) as well as considering the kind of questions we get here at Acquia from companies considering adopting Drupal. In terms of past SAs, cross-site scripting (XSS) is the most commonly found vulnerability in Drupal core and contributed modules and themes.
1. Twig templates used for html generation
This is probably first on the list of anyone you ask about Drupal 8 security. This is also one of the most popular features with themers.
One security gain from this is that it enforces much stricter separation of business logic and presentation – this makes it easier to validate 3rd party themes or delegate pure presentation work. You can't run SQL queries or access the Drupal API from Twig.
In addition, Drupal 8 enables Twig auto-escaping, which means that any string that has not specifically flagged as safe will be escaped using the PHP function htmlspecialchars() (e.g. the same as Drupal 7 check_plain()). Auto-escaping of variables will prevent many XSS vulnerabilities that are accidentally introduced in custom site themes and custom and contributed modules. That fact is why I ranked this as number one. XSS is the most frequent security vulnerability found in Drupal code. We don't have a lot of hard data, but based on past site audits we generally assume that 90% of site-specific vulnerabilities are in the custom theme.
2. Removed PHP input filter and the use of PHP as a configuration import format
OK, maybe this should have been number one. Drupal 8 does not include the PHP input format in core. In addition to encouraging best practices (managing code in a revision control system like git), this means that Drupal no longer makes it trivial to escalate an administrator login to being able to execute arbitrary PHP code or shell commands on the server.
For Drupal 7, importing something like a View required importing executable PHP code, and for certain custom block visibility settings, etc. you would need to enter a PHP snippet. These uses of evaluated PHP (exposing possible code execution vulnerabilities) are all gone – see the next point about configuration management.
Now that we have covered the top two, the rest of the 10 are in rather arbitrary order.
3. Site configuration exportable, manageable as code, and versionable
The Configuration Management Initiative (CMI) transformed how Drupal 8 manages things that would have been represented in Drupal 7 as PHP code. Things like Drupal variables or ctools exportables (e.g. exported Views).
CMI uses YAML as the export and import format and the YAML files can be managed together with your code and checked into a revision control system (like git).
Why is this a security enhancement? Well, in addition to removing the use of PHP code as an import format (and hence possible code execution vulnerability), tracking configuration in code makes it much easier to have an auditable history of configuration changes. This will make Drupal more appealing and suitable for enterprises that need strict controls on configuration changes in place. In addition, configuration can be fully tested in development and then exactly replicated to production at the same time as any corresponding code changes (avoiding mistakes during manual configuration).
Finally, it is possible to completely block configuration changes in production to force deployment of changes as code.
4. User content entry and filtering improved
While the integration of a WYSIWYG editor with Drupal core is a big usability improvement, extra care was taken that to mitigate poor practices that adding a WYSIWYG editor encouraged in past Drupal versions. In particular, users with access to the editor were often granted access to the full html text format, which effectively allowed them to execute XSS attacks on any other site user.
To encourage the best practice of only allowing the use of the filtered HTML format, the Drupal 8 WYSIWYG editor configuration is integrated with the corresponding text filter. When a button is added to the active configuration, the corresponding HTML tag is added to the allowed list for the text filter.
Drag a new button from the available to enabled section in the editor configuration: WYSIWYG editor configuration adding underline button The corresponding HTML tag (the U tag) is added to the allowed list: U tag is allowed in the filter An additional security improvement is that the core text filtering supports limiting users to using only images local to the site which helps prevent cross-site request forgery (CSRF) and other attacks or abuses using images.
5. Hardened user session and session ID handling
There are three distinct improvements to session and session cookie handling. First, the security of session IDs has been greatly improved against exposure via database backups or SQL injection (7.x back-port ). Previously in Drupal, the session ID is stored and checked directly against the incoming session cookie from the browser. The risk from this is that the value from the database can be used to populate the cookie in the browser and thus assume the session and identity of any user who has a valid session in the database. In Drupal 8, the ID is hashed before storage, which prevents the database value from being used to assume a user's session, but the incoming value from the value is simply hashed in order to verify the value.
Next, mixed-mode SSL session support was added to core to support sites that, for example, used contributed modules to serve the login page over SSL while other pages unencrypted. You will have to replace the session handling service if you really need this. This encourages serving your entire site over SSL (which is also a search engine ranking boost).
The final change is that the leading “www.” is no longer stripped from the session cookie domain since that causes the session cookie to be sent to all subdomains (7.x back-port).
6. Automated CSRF token protection in route definitions
Links (GET requests) that cause some destructive action or configuration change need to be protected from CSRF, usually with a user-specific token in the query string that is checked before carrying out the action.
This change improves the developer experience and security by automating a process frequently forgotten or done incorrectly in contributed modules. In addition, centralizing the code makes it easier to audit and provide test coverage. Drupal 8 makes it easy. A developer merely needs to specify that a route (a system path in Drupal 7 terms) require a CSRF token. Here is an example of the YAML route definition for a protected link in Drupal 8 entity. entity.shortcut.link_delete_inline: path: '/admin/config/user-interface/shortcut/link/{shortcut}/delete-inline' defaults: _controller: 'Drupal\shortcut\Controller\ShortcutController::deleteShortcutLinkInline' requirements: _entity_access: 'shortcut.delete' _csrf_token: 'TRUE' Only the one line in the requirements: section needs to be added to protect shortcut deletion from CSRF.
7. Trusted host patterns enforced for requests
Many Drupal sites will respond to a page request using an arbitrary host header sent to the correct IP address. This can lead to cache poisoning, bogus site emails, bogus password recovery links, and other problems with security implications. For earlier versions of Drupal, it can be a challenge to correctly configure the webserver for a single site that uses sites/default as its site directory to prevent these host header spoofing attacks. Drupal 8 ships with a simple facility to configure expected host patterns in settings.php and warns you in the site status report if it's not configured.
8. PDO MySQL limited to executing single statements
If available, Drupal 8 will set a flag that limits PHP to sending only a single SQL statement at a time when using MySQL. This change would have reduced the severity of SA-CORE-2014-005 (a SQL injection vulnerability that was easily exploited by anonymous users) (7.x back-port)
. Getting this change into Drupal 8 meant I first had to contribute a small upstream change to the PHP language itself, and to the PDO MySQL library that is available in PHP versions 5.5.21 or 5.6.5 and greater. There is also a patch in progress to try to enforce this protection regardless of which specific database driver is being used.
9. Clickjacking protection enabled by default
A small change, but Drupal 8 sends the X-Frame-Options: SAMEORIGIN header in all responses by default. This header is respected by most browsers and prevents the site from being served inside an iframe on another domain. This blocks so-called click-jacking attacks (e.g. forms or links on the site being presented in a disguised fashion on an attacker's site inside an iframe), as well as blocking the unauthorized re-use of site content via iframes. (7.x back-port).
10. Core JavaScript API Compatible with CSP
Support for inline JavaScript was removed from the #attached property in the Drupal render API. In addition, the Drupal javascript settings variables are now added to the page as JSON data and loaded into a variable instead of being rendered as inline JavaScript. This was the last use of inline JavaScript by Drupal 8 core, and means that site builders can much more easily enable a strict content security policy (CSP) – a new web standard for communicating per-site restrictions to browsers and mitigating XSS and other vulnerabilities. A final note of caution: The substantial code reorganization and refactoring in Drupal 8 as well as the dependence on third party PHP components does present a certain added risk. The code reorganization may have introduced bugs that were missed by the existing core tests. The third party components themselves may have security vulnerabilities that affect Drupal, and at the very least, we need to track and stay up to date with them and fix our integration for any corresponding API changes. In order to try to mitigate the risk, the Drupal Association has been conducting the first Drupal security bug bounty that has been run for any version of Drupal core. This has uncovered several security bugs and means they will be fixed before Drupal 8 is released.
- Source: https://goo.gl/i2CCxj
Adrian Ababei / Oct 23'2015
The concept of Modular CSS has been around for years. The basic idea is tackling your CSS code in a systematic way through class and file naming conventions. The main goal being improved workflow through organization, readability and reusability. Some of the more recent modular CSS architectures are Object Oriented CSS (OOCSS) and Scalable and Modular Architecture for CSS (SMACSS). I tend to prefer SMACCS as Jonathan Snook has presented it as a set of guidelines rather than rules. He leaves it open for individual style and interpretation as every developer and every project is different. If you want to see a great example of Modular CSS in action, go blow your mind looking at the Twitter Bootstrap Components Library. The creators have done an excellent job abstracting the components in a way that makes them easy to understand and flexible via classes and subclasses. The idea of Modular CSS really appeals to me, especially having worked on large complex sites with multiple front-end developers collaborating together. While Drupal is perfect for these large scale projects, adding your own CSS classes to a Drupal theme ( essential when working with modular CSS ) is often unintuitive, inconsistent and tiresome at best. To remedy this, I've been working on a systematic approach to adding classes to themes that has so far proved extremely useful. In this post we're going to see how we can easily manage classes on one of Drupal's most used components, Blocks, using a preprocess function. Preprocess Functions Preprocess ( and Process ) Functions are the key to taking control of your classes in Drupal. Preprocess functions are called before every theme hook is sent to a theme function and template file (.tpl). In my mind preprocess functions have 2 uses: Altering and adding variables that are available to theme functions and .tpl's. This is how we'll be adding our classes. Sending these variables to an alternate theme function or .tpl via theme hook suggestions. I prefer managing my classes via preprocess functions over template files because I find it way more flexible and concise. I no longer need duplicate template files just to add or remove a single class. Adding Classes to Blocks As is often the case with Drupal, there are a few options out there for adding classes to blocks, including Skinr, Fusion Accelerator and Block Class. All of these offer the power of adding classes through the admin UI which may be right for you and your site administrators. This hasn't been a requirement for most sites I've worked on. Your mileage may vary. I prefer to define my classes directly in the theme where I'm already writing my CSS, everything is already in code and there is no need to export settings from the database. So let's see how we can do this using default core functionality. Looking at the core block.tpl.php file, by default there are 3 elements to which we can add classes during preprocess: the block container itself ($classes), the block title ($title_attributes) and block content ($content_attributes). As of this writing, in order to preprocess the block content classes, you must remove the hard coded class attribute from the template yourself or apply a core patch.
Now let's create our preprocess function and start adding some classes.
This may look like a lot but it's actually pretty simple. Let's break it down. First off, we create some shortcut variables to save us some typing down the road and make our code more legible. The first variable we create is $block_id which is just a string combining the name of the module creating the block and the block's delta— separated by a dash. This gives us an easy way to target individual blocks when we start adding classes. One thing to note, some blocks, including core custom blocks, use an auto-incremented integer for their delta value. This can present issues if you are moving blocks from dev to staging to production and you're not careful. I recommend using the boxes module to help mitigate this. The next three variables, $classes, $title_classes and $content_classes are references to the arrays that will eventually be rendered into class attributes. To add classes to an element, we just need to add them to its corresponding array. After we have our shortcuts set, we can started adding classes to our blocks. We first start by adding some global classes, .block-title and .block-content, to all block titles and block content containers respectively. You may or may not find this helpful. If you're interested in stripping out all the classes that core and other modules have added to your blocks up to this point, this would be the place to do it. To remove those classes, just reset your variable to an empty array like so $classes = array(); Next, we can start adding our classes on a per block basis. I like to keep a simple print statement handy that prints out the $block_id for all blocks on the page. When styling a new block, I just uncomment this line and copy the id from the browser into my code editor. To target individual blocks, we create a simple switch statement based on the $block_id. For each block we need to style, we write a case and add the corresponding classes there. In the first case we are adding three classes to the System Navigation block, one to the block itself, one to its title element and one to its content wrapper. In the second instance we are actually grouping two cases together so we can add the same classes to multiple blocks. Here we are adding .element-invisible, Drupal's built in class for hiding content in an accessible manner, to the titles of the Main Menu and User Login Blocks. As we add blocks, we add more switch statements. That's all we really need to do to take over our block classes. I've found this alone can be a tremendous boost to productivity while styling a Drupal site. You can get pretty far with blocks alone. However, with modifications to core theme functions and templates, similar to what we did with block.tpl.php above, this technique can be adapted to manage classes on nodes, fields, menus and most other themeable output. I've submitted a session proposal to DrupalCon Munich on this topic called Stay Classy Drupal - Theming with Modular CSS. If you're interested in learning more about taking control of your classes or have run into trouble trying to do so, leave a comment to let the organizers know you're interested in the session.
Adrian Ababei / May 06'2015
Microsoft is equipping its Azure cloud service with a tool to debug PHP-based Web applications that are running on the platform. The service “offers in-depth insight into how an application is performing,” said Andi Gutmans, who is the CEO of Zend Technologies and one of the original developers of PHP.
Zend is supplying the debugging tool, called Z-Ray. The software “exposes a huge amount of information to the developer through the Web browser,” he said. PHP is one of the most widely used languages for building Web applications and dynamic Web sites.
Microsoft has long supported PHP on its Azure platform, offering a PHP runtime environment and a selection of PHP aids such as a software development kit and a lightweight integrated development environment called WebMatrix. The new additional debugging capabilities indicate Microsoft’s interest in staying competitive with other cloud providers in supporting Web applications, one of the primary uses of cloud computing services today.
As a Web programming language, PHP code can be particularly thorny to debug. When something goes wrong, error messages may not make it to the browser screen, leaving developers scratching their heads. Zend designed Z-Ray to give developers more insight into how their PHP code is operating under the hood of the browser. It details each PHP to call the browser makes to the server and reveals the code making those calls. The software, for instance, can show what specific database queries PHP has issued, and how long they are taking to complete. It can show how the session data changes between user requests.
In addition to revealing the inner workings of PHP, Z-Ray can also be used to troubleshoot many popular Web platforms written in PHP, such as the WordPress and Drupal blogging software and the Magento e-commerce platform. “It can show the architectural issues that are otherwise almost impossible to figure out,” Gutmans said. The Z-Ray service will be available as an optional add-on for PHP Azure users in two editions, a basic version available without charge and a more comprehensive version available for a “modest,” still undetermined, fee, Gutmans said. The service will be available as a preview after Microsoft’s Build developer conference at the end of this month.
Zend, which bills itself as “the PHP company,” also offers Z-Ray as stand-alone software, and as a service on the Amazon Web Services (AWS) marketplace.
Source: https://goo.gl/Gj5e2V
Adrian Ababei / Apr 17'2015
Now Google is automatically converting Adobe Flash ads directly to HTML5. What this means is that it’s much easier for advertising campaigns to target users within the Google Display Network without needing a browser or device that supports Flash.
From Flash to HTML5
In September, when Flash has stopped being supported, Google started to offer HTML5 backups or simply put Flash to HTML5 conversion tools to its Google Display Network. DoubleClick Campaign Manager created a HTML5 version of all the Flash ads in use which preserved Flash’s effects, look and feel.
Nowadays Google is simply converting Flash campaigns into HTML5 automatically. Campaign managers just need to upload their ads with AdWords Editor, AdWords or any other tool that’s supported with Google’s platform.
Google mentions certain limitations on the Image ads support page, under the “How to determine if a Flash ad is running as an HTML5 ad” section:
Not all Flash ads can be converted to HTML5. To see whether your Flash ad will convert, upload it to the Swiffy tool. If this tool is able to convert your ad, then your ad will be automatically converted when it is uploaded to AdWords.
To see in your AdWords reports whether your Flash ad was converted, segment your ad table by devices. If you see mobile or tablet impressions for a Flash ad, then your ad was converted.
In the future, we will be providing a notification on all converted Flash ads. We expect this to be available in late 2015.
The death of Flash
Basically, Google is working hard to convert all ads from Flash to HTML5 but not all ads are eligible for the conversion, not just yet anyway. The tech giant wants to switch to HTML5 entirely, creating all ads in HTML5 from the beginning.
The trend of ditching Flash in favour of HTML5 is not a new thing – Google has been pushing for it since January 2010 when YouTube announced it has been testing with a HTML5-based video player.
Adrian Ababei / Mar 04'2015
At OPTASY we’re big fans of Drupal. While we remain a technology agnostic firm and believe in matching organization needs with the best technology available, we believe Drupal is an excellent platform that addresses a wide range of web requirements. Now, with the recent beta version release of Drupal 8, the future of Drupal is incredibly bright.
Drupal 8, which is currently in beta mode and slated for full production release sometime in 2015, represents a large-scale reimagining of the core Drupal application. The result of this grand effort is a much more cohesive, flexible and integrated platform on which to build websites of the future.
Some folks have come to us and are already considering the move to Drupal 8 and are interested in becoming early adopters. However, before we get into specifics about Drupal 8, we want to highlight two key reasons we will likely continue to build on Drupal 7 for all of 2015 and into 2016:
The true power of Drupal lies in the community modules that are available, and it will be at least 3-6 months until a critical mass have been upgraded and are production ready for Drupal 8. This timeline starts once Drupal 8 is released into production.
In community development right now are a set of migration tools that should significantly ease the process of migrating from Drupal 7 to 8 at the appropriate time. This significantly reduces the reason to build on Drupal 8 out of the gate, and the majority of sites to start using 8 are likely to have been migrated from 7.
We would not support Drupal 7 if it was not a best-in-class application. That said, in building websites on 7, we have spent much time over the years on activities such as turning off unneeded features, applying a group of our favorite modules to streamline and improve the admin interface, and tieing in an array of symbiotic third-party tools for caching and performance. While this knowledge is great for an agency that works with Drupal constantly, it can be harder for non-expert Drupal users to get up to speed. The inclusion of many strategies we use for Drupal 7 natively into Drupal 8 will be great for getting sites up to speed quickly. We have also written about some of our other Drupal implementation best practices which may be interesting to those looking for a more technical experience with Drupal.
Drupal 8 Features We Love
1. Manage your site's content...from your phone. Anyone that’s ever administered a website knows that it’s best done while using a desktop or laptop. Much of the backend functionality simply doesn’t work, or doesn’t support mobile platforms. Enter Drupal 8, which is being built from the ground up with a focus on multi-device support. The core themes available for Drupal 8 will utilize responsive design, on both the front and back-end. Not only will visitors to your site have an optimized viewing experience for every device, your administrators will be able to manage the site from mobile devices as well.
2. Add new features to your live site with ease. Prior versions of Drupal store virtually all configuration settings in the Drupal database, alongside all site content (such as pages, menus, blocks, metadata and users). This approach greatly complicates deploying your updated site (e.g. code with new features) to your live site, as you can’t simply overwrite the live database without erasing your latest content and user activity in the process. In Drupal 7 we use the Features and Strongarm modules to build exportable packages of settings, Drupal 8 now stores these settings in files found in your library directory. This greatly simplifies deploying configuration changes such as new content types, fields or views from development to production.
3. Improved performance capabilities through modernization. As part of the overall transition to the Symfony framework, Drupal 8 offers an alternative to the unique "hook" system for attaching modules to Drupal core. Through the Symfony Event Dispatcher, application components can communicate with one another, allowing the system as a whole to run much more intelligently. What this means for the end user is that unlike the default with prior versions of Drupal, the system does not need to load every enabled module with each page request. And as many core Drupal hooks have been replaced, Drupal as an application is well on its way to potentially replacing hooks altogether in Drupal 9.
4. Integration capabilities significantly improved. Already one of the most flexible platforms when it comes to integrating with third-party applications, Drupal 8 takes the possibilities to an entirely different level. Using the new Rest & Serialization APIs, site builders will be able to output serialized data as JSON and XML from Drupal, almost as easily as they can normal HTML output.
5. Take your website around the world. Drupal 7 supports multi-lingual functionality, but not at the level that Drupal 8 will. Imagine translating your site’s content to any one of the 110 different supported languages with a few clicks. You’ll be able to not only translate a page’s content for a specific language, but also build views and determine what blocks should appear for that language. Translation updates will also be pushed to the site automatically to make sure your site has the latest dictionary.
Why We Love Drupal in General
1. Site management made easy with Drupal content types. One of the keys to getting the most benefit from a CMS lies in the separation of content from design and layout. The greater extent to which you can isolate these two site components, the more your graphic designers and content authors/editors can be freed to do their jobs without interference from one another. Drupal bakes this concept into its very core by making it quite easy to create numerous individual content types, each with its own collection of fields and possible field types. Gone are the days where authors are presented a blank slate for each page via a single rich text area. With Drupal, your designers can easily ensure these fields get styled correctly and displayed in the right orientation, while authors are presented with simple fields to fill out and edit.
2. Scalability and flexibility galore. Drupal is one of the most flexible platforms on the market right now and allows virtually unlimited customization in the front-end, backend, and everything in-between. The key lies in the thousands of excellent off-the-shelf modules available at drupal.org. While some may require customization, they can be added on to core Drupal with relative ease. This ensures your site can grow and change as your organization does the same.
3. System administrator automation via Drush. Freely available for download and immediate use on all Drupal sites, Drush provides an awesome command-driven interface for managing Drupal. Admin tasks that require many clicks and page refreshes are easily reduced to a single command. For example, we find Drush most useful in the creation of new user accounts (especially if you need to create many at once), generation of system backups, application of core and module updates, and migration of content from other systems.
4. Robust security and overall support options. Software companies typically have dozens, or at most hundreds of developers working on support, bug fixes, new features, integrations. The Drupal community comprises over 180,000 active contributors. This facilitates rapid advancement for Drupal and establishes a very large population of modules and integrations.
Adrian Ababei / Feb 24'2015
Wordpress to Drupal 8 migration setup
Here’s an example of an existing WordPress site, which has registered users with comments, anonymous comments and an archive of previous postings with metadata (tags) that we will import. We’ll finish with this posting displayed in Drupal 8 version of the blog.
The following tables in Wordpress are relevant to the migration:
wp_posts (contains blog posts and pages)
wp_comments (comments)
wp_users (users)
wp_terms (categories - taxonomy)
We will started writing our migration module using Migration API (migrate) that comes packaged with D8 core. As usual in Drupal 8 module development, we had .info.yml, files, which uses the Symfony 2 YAML to map data types into an XML-like schema for easy migration from one environment to another, and traditional Drupal .module files. The only thing worth noting here is a dependency on the migrate module in our info.yml file.
For each Entity migration (users, posts, comments) we created 2 more files:
A YAML file containing ID, Source, Destination and Mapping (Process). Put this under MODULE_ROOT/config/install/
A Source Plugin used to fetch/process Wordpress data. Put this underMODULE_ROOT/src/Plugin/migrate/source/
Code breakdown
Users
Let start understanding things by migrating Users first. Here is the complete user YAML file for this migration. Let’s break it down:
In this migrate.migration.users.yml file, ID, Source, Destination and Process (field mapping) are important keys.
id: users # Links this file with wordpress.migrate.yaml
label: Wordpress Users
migration_groups:
- Wordpress
ID key is ‘users’ which has been referred in manifest_wordpress.yml and also in Source Plugin (User.php) as part of Annotation.
source:
plugin: users
Source key is referring to Source Plugin (User.php) created by us to fetch Wordpress users data, which we will see shortly in detail.
destination:
plugin: entity:user
Destination key is user entity as defined by Migration API.
process:
uid: id
name: user_login
pass: user_pass
mail: user_email
status:
-
plugin: default_value
default_value: 1
created:
-
plugin: callback
callable: strtotime
source: user_registered
Process is key to mapping source (Wordpress) and destination (Drupal) fields. Some fields are mapped directly like ‘id’ with ‘sid’ while some fields are pre-processed by Migration API before being assigned to a destination field. For instance, ‘user_registered’ will be processed by PHP function strtotime() before it’s value gets assigned to destination field, because the target Drupal site must recognize those users as registered to associate their previous activity in WordPress with their Drupal activity on the new site. We can even write our own process plugins if required.
Now let’s look at Source Plugin (Users.php) which has been used to fetch the Wordpress users data,
<?php
/**
* @file
* Contains \Drupal\migrate_wordpress\Plugin\migrate\source\Users.
*/
namespace Drupal\migrate_wordpress\Plugin\migrate\source;
use Drupal\migrate\Row;
use Drupal\migrate\Plugin\SourceEntityInterface;
use Drupal\migrate_drupal\Plugin\migrate\source\DrupalSqlBase;
/**
* Extracts users from Wordpress database.
*
* @MigrateSource(
* id = "users"
* )
*/
Remember the id field in the YAML file? The id field tells Drupal that this class should be used to process data for that migration set, taking the results of “users” in WordPress and adding them to a new table in the Drupal database, wp_users.
class Users extends DrupalSqlBase implements SourceEntityInterface {
/**
* {@inheritdoc}
*/
public function query() {
return $this->select('wp_users', 'u')
->fields('u', array_keys($this->userFields()));
}
The query() method is used to fetch records from Wordpress database table (wp_users).
/**
* Returns the User fields to be migrated.
*
* @return array
* Associative array having field name as key and description as value.
*/
protected function userFields() {
$fields = array(
'id' => $this->t('User ID'),
'user_login' => $this->t('Username'),
'user_pass' => $this->t('Password'),
'user_email' => $this->t('Email address'),
'user_registered' => $this->t('Created time'),
);
return $fields;
}
/**
* {@inheritdoc}
*/
public function fields() {
$fields = $this->userFields();
}
userFields() method is called in query() and fields() methods. It contains the list of fields that describe a WordPress user along with their field descriptions.
/**
* {@inheritdoc}
*/
public function bundleMigrationRequired() {
return false;
}
/**
* {@inheritdoc}
*/
public function entityTypeId() {
return 'users';
}
/**
* {@inheritdoc}
*/
public function getIds() {
return array(
'id' => array(
'type' => 'integer',
'alias' => 'u',
),
);
}
As we are also implementing an interface SourceEntityInterface, we need to define 2 more methods namelybundleMigrationRequired() and entityTypeId() to map user-generated content, such as postings and comments, to Drupal. The getIds() method is used to return the key field for this Entity.
Posts
The Posts Entity migration (YAML, Source Plugin) is quite similar to User Entity migration. There is one additional method prepareRow() defined in post migration source plugin. This method gets called for each row to do an additional processing of the source data.
public function prepareRow(Row $row) {
$post_type = $row->getSourceProperty('post_type');
$type = $post_type == 'page' ? 'page' : 'article';
$row->setSourceProperty('type', $type);
return parent::prepareRow($row);
}
In this method, we are first checking the post type of source data and based on that we set the ‘type’ property. As we have two types of posts in Wordpress (Pages, Blogs posts) we want to map them to two content types in Drupal (Basic page, Article). The ‘type’ property is available in the post-migration YAML file Process step, as discussed above, and we have mapped it to the Drupal content type field.
Comments
The Comments Entity migration is more or less similar to Users and Posts migration. There are some following additions in Comments migration YAML file which are quite interesting to know.
uid:
-
plugin: skip_process_on_empty
source: user_id
-
plugin: migration
migration: users
'comment_body/value': comment_content
'comment_body/format': 'basic_html'
migration_dependencies:
required:
- posts
- users
Here we are using ‘skip_process_on_empty’ plugin which means skip the process if the source value is empty. As key fields can have different values in source and destination databases so we are using migration plugin to map ‘user_id’ field with ‘uid’ field. This will look for destination id in migration map tables for given source id (user_id).
Next, the ‘comment_body’ field is a text field and made up of two fields in Drupal 8, so we must map the Wordpress ‘comment_content’ field to value field and mapping basic html to format field. Next we have defined comment migration dependencies to posts and users which means comments will be migrated once both posts and users are already migrated.
Putting it all together
Finally you can run the migration using 'migrate-manifest' drush command as below. Make sure to run this command in Drush 7.
drush migrate-manifest manifest_wordpress.yml --legacy-db-url=mysql://{dbuser}:{dbpass}@localhost/{dbname}
The manifest_wordpress.yml is a YAML file which contains reference to IDs of each migration (users, posts, comments etc.).
# A Wordpress posts/users/comments migration, with dependencies.
- posts
- users
- comments
Finished: WordPress postings shown in Drupal
Adrian Ababei / Feb 24'2015
This feature has long been a goal for OPTASY; it’s been more challenging than we expected. You see, automation is easy if all sites are built from the same base repository using the same folder structure and implementing the same base modules. Yet OPTASY supports fabulous client sites that, because they leverage the complexity and extensibility of Drupal they are creative sites., each unique, blending modules, custom code and content. A generic one-size-fits-all update process has never worked.
In the past, we have manually updated sites using local scripts and patches, but this method was never going to succeed in providing security updates within 24 to 48 hours of a security release.
Early on, we decided that well-built automation needs to:
Identify and provide security updates released on drupal.org in a timely manner.
Implement a clear path of deployment and testing which starts from and, after validating each environment (Dev; Staging, and Production), and then returns to retest production site code to ensure all updated components have been propagated across all environments.
Allow for testing independently of ongoing development by the customer.
Provide clear and consistent communication to all stakeholders about pending changes.
RA Automation has (mostly) arrived!
Finally, we are well on our way to achieving exactly those standards, providing automated security updates to RA subscriptions within 48 hours of a security release, to both the core and contributed modules. We expect to reach this goal in the first half of 2015, but the beta functionality is already updating RA subscriptions.
With the introduction of automated patch and update deployment, there are minor but meaningful changes going on with some of the features you are familiar with today.
RA Environment
In order to avoid interrupting ongoing client development, OPTASY provides a free RA environment. For OPTASY Cloud Enterprise (ACE) clients, the RA environment is on a completely separate server to ensure that RA deployment and testing does not overburden customers’ dedicated testing environments. The redundant server allows security updates to be deployed into client environments as soon as they are available and readied for client testing.
RA Preferences
Last year, client-facing RA preferences were made available in your RA administrative settings. Clients can control RA automation on a per-subscription level. We plan to continue to refine these to work best with the script and the unique needs of clients.
Using APIs
OPTASY RA has utilized the internal OPTASY Cloud API in order to take advantage of the tools built by our fabulous cloud team. The API allows automation to read RA Preferences as well as other centralized subscription data that is used to create human-readable tickets and monitor progress. Optasy RA also takes advantage of the Zendesk API, and has contributed improvements back to Zendesk.
Drush and LiveDev
Since automated updates use Drush, it takes advantage of the Cloud Live Development feature. All updates take place on the RA Environment first, then are switched to Live Development for the update and code commits that will be deployed for the particular customer.
While drush typically does all updates at once, we ensure that each module update is a discrete git or svn commit. This allows much more efficient troubleshooting down the line, as load order can impact how the whole instance works due to custom code. The additional staging step, in the RA Environment, allows OPTASY to revert a single module update. It also allows customer development teams to cherry-pick the updates that best fit into their ongoing development process.
Drush Backend
OPTASY has also contributed back to the Drupal community by sponsoring development in Drush, eliminating a problem that interfered with automated updating.
RA clients will know that enabling the Update module is essential in order to get security updates. Drush 6 requires this module to access module status information without which Support teams have no way of knowing the status of a site’s modules. Asking clients to enable the module so we could scan always took extra time, often delayed the process.
In Drush 7 the Update module unnecessary. The fabulous-to-work-with F.L. Jonathan Araña Cruz developed the code, while Samuel Mortenson tested on it OPTASY servers. The result is the ‘Drush Backend’ which has been committed to the current dev version of Drush 7.
This work has streamlined initial versions of our automation, removing a significant amount of code that worked around Update cache issues (pm-enable update, pm-refresh, truncate cache_update, etc.). We reduced the total number of pm-updatecode calls. This also means that enabling the update module is no longer a requirement for RA subscriptions.
Automation now and in the future
Currently, OPTASY automation requires manual initiation via a queuing system. The OPTASY RA team always initiates the queue when a core security update is released. Otherwise, we initiate a periodic scan. Based on the improvements described here, RA customers can look forward to the following:
Daily automated scanning which will create a new branch every two weeks, so as not to flood you with new updates.
The ability to approve and automatically initiate all stages in the update process. This means that, unless you need troubleshooting from our team first, you can apply a security update without any required intervention by OPTASY.
The ability to specify bug-fix updates that can be included in a security update, or done independently. This will be a feature of RA Premier only.
Support for patching via Drush make files.
Finally, since Automation is under active development, this is a great time to let us know what features and capabilities you would like! Please feel free to comment below, I look forward to hearing your thoughts.
Adrian Ababei / Feb 24'2015
This tutorial covers writing a "Hello World" test for Drupal 7 using the SimpleTest framework that comes with Drupal core. Knowing how to write tests is an important skill, and being able to do so can help to increase the quality of your code, and save you hours of time clicking around your site making sure things still work after making changes.
For example, we've got a feature on our site that allows anyone to embed a copy of any of our free videos into another site. It's a great feature, but to be honest, it's also one that we forget about. It's hidden beneath the "share" link on video pages, and it's not something that any of us do very often. Whenever we make changes to our video player we're diligent about testing them on our site, but often times forget to check the embed codes. Testing them is also a bit of a tedious process because you need to find a free video, find the embed code, copy it into an HTML file on another site, and then test it. As such, we tend to break it from time to time without realizing we've done so until much later.
After having done this a couple of times though, we added test coverage for it. So now, whenever we make changes to any part of the site, one of the things that happens before that change is made live is we subject a copy of the site with the changes applied to a barrage of automated tests. If we break that embed code again (which we have), we'll know about it before it becomes a problem on the live site.
This is just one of the many useful things you can do with automated tests. Keep reading to learn how to write your first test.
After completing this tutorial you should be able to:
Create a new PHP class that defines a test
Provide Drupal with meta-data about your test
Run your test to see if it works
Background
This tutorial assumes that you are familiar with PHP and basic OOP concepts, and that you've written a Drupal module before and know how to get Drupal to recognize your module and allow it to be enabled.
Hello World
Start by creating a simple helloworld module directory in sites/all/modules/custom/helloworld/
Add a helloworld.info file with the following contents:
name = Hello World
description = An example module to demonstrate using SimpleTest
core = 7.x
Add a helloworld.module file with the following contents:
<?php
/**
* @file
* Hello World demonstrates the use of SimpleTest for Drupal 7.
*/
/**
* Implements hook_menu().
*/
function helloworld_menu() {
$items = array();
$items['helloworld'] = array(
'title' => 'Hello World',
'access callback' => TRUE,
'page callback' => 'helloworld_hello_page',
'type' => MENU_NORMAL_ITEM,
'menu' => 'navigation',
);
return $items;
}
/**
* Page callback for /helloworld.
*/
function helloworld_hello_page() {
return t('Hello World. Welcome to Drupal.');
}
In the Administration menu, go to Modules (admin/modules).
Enable the Hello World module.
In the navigation menu navigate to Hello World (helloworld), where you should now see the text "Hello World: Welcome to Drupal." displayed on the page.
This is what we're going to write a test for. Ensuring that when our module is enabled, a user can navigate to the page /helloworld and see this text. Let's get started.
Extending DrupalWebTestCase
Any module can optionally contain one or more test cases, and those test cases can either all be in a single file, or spread out among many files. The convention is for files that contain tests to end with the .test extension. And if you've only got one, to name it MYMODULE.test.
Start by creating a new file: helloworld/tests/helloworld.test, we will add the code that makes up our test case to this file.
All functional tests in Drupal should extend the DrupalWebTestCase class which will provide the utility functions used to drive the test browser, helper functions for interacting with a Drupal application, and all the assertions we will use to perform our actual tests.
Add the following to your helloworld.test file:
<?php
/**
* @file
* Tests for the hello world module.
*/
class HelloworldTests extends DrupalWebTestCase {
}
Add a getInfo() method to your HelloworldTests class. This method returns an array that provides meta-data about our test that the SimpleTest module can use when displaying our test in the UI. This method is required in order for your test to work.
/**
* Metadata about our test case.
*/
public static function getInfo() {
return array(
// The human readable name of the test case.
'name' => 'Hello World',
// A short description of the tests this case performs.
'description' => 'Tests for the Hello World module.',
// The group that this case belongs too, used for grouping tests together
// in the UI.
'group' => 'Hello World Group',
);
}
Add a setUp() method to HelloworldTests. This method is called when the system is preparing the environment for running our tests. It allows us to perform any setup steps required. In this case, we need to ensure that our module is enable during setup so that the page we want to test is available. We can do this by calling the parent classes's setUp() method and passing it an array of modules we want enabled.
/**
* Perform any setup tasks for our test case.
*/
public function setUp() {
parent::setUp(array('helloworld'));
}
Add a test*() method to our class. Each test case can have one or more methods whose names are prefixed with the string test. Each of these methods will be automatically called by the SimpleTest test runner, and should contain the various assertions required in order to demonstrate that this test is passing. This example will use the DrupalWebTestCase::drupalGet() method in order to tell the internal browser to navigate to the URL /helloworld. And then use the DrupalWebTestCase::assertText() method to verify that the string, "Hello World ...", exists somewhere in the HTML output of the page the internal browser is currently looking at.
public function testHelloWorld() {
// Navigate to /helloworld.
$this->drupalGet('helloworld');
// Verify that the text "Hello World ...", exists on the page.
$this->assertText('Hello World. Welcome to Drupal.', 'The page content is present.');
}
Finally, we need to tell Drupal about our new test. Update your helloworld.info file and add the following line so that after clearing the cache the registry will contain a reference to our new HelloworldTests class.
files[] = tests/helloworld.test
Run your tests
Clear the cache so Drupal locates our new test(s). Navigate to Configuration > Performance (admin/config/development/performance) in the Toolbar, and then clicking the "Clear all caches" button.
Navigate to Modules (admin/modules) in the Toolbar. Enable the Testing module in the Core section.
Navigate to Configuration > Testing (admin/config/development/testing).
Select the checkbox for the "Hello World" test to run all tests in this group, or alternately tip the drop down open and choose the individual test case you want to run.
Click the "Run tests" button.
A new page will open and display a progress bar while your tests run.
Once completed, you will be directed to a page with a table that shows the results of your tests.
Summary
In this tutorial we've written a Hello World test in order to demonstrate some of the fundamental concepts necessary for writing tests for Drupal with SimpleTest. Including, defining our HelloworldTests class in a .test file and making sure Drupal can find it by adding it to our module's .info file. Then providing meta-data about our tests for the SimpleTest UI, and any setup instructions necessary so that the environment in which our tests run is appropriate for our test case. Finally, we wrote a test*() method that used the internal SimpleTest browser to navigate to a page and verify that a string of text is present on that page.
I encourage you to explore SimpleTest through the Testing module in Drupal, and the concept of automated testing further as it provides numerous sanity and time saving benefits. Enjoy.
Source: https://goo.gl/MGzNMi
Adrian Ababei / Feb 21'2015