Drupal 7: 32% faster online store page generation

Page load speed affects conversion and indexing by Google and Yandex. In some known cases boosting page load yielded a 40% increase in the number of orders or 13% greater revenue. Optimization of Drupal-powered websites and servers hosting them to make pages load faster is a routine task for our team.

Page generation time (by web server) is one of the key metrics in the overall site loading speed. Site and server optimization with the aim to boost it is a joint effort that employs our system administrators and DevOps specialists, developers, specialists from the customer's and hosting provider's side. Searching for and eliminating bottlenecks in the site's code, we profile slow pages with XDebug and analyze runtime of individual PHP functions. NewRelic monitoring service allows us to collect aggregated stats on the page request processing time. Finally, we analyze the statistics of MySQL and Solr query execution.

Below you will find an account of one of the cases we solved, a 32% online store page generation boost through remedying a problem common to Drupal-powered websites.

The problem

Drupal 7 has a longstanding "feature": when AJAX is used, the forms found on the page are cached automatically. Each output of such a form creates two entries in the cache_form table, one describing its structure and the other its state. Correct operation of the AJAX handler requires caching, since it needs to know the form's structure and its last state.

When a page contains many forms, the number of entries in the cache_form table grows rapidly, which is a problem. The situation becomes dire when an online store running Commerce makes use of an AJAX-powered Add to Cart button. For example, 50 products on a catalog page yield 100 entries in the cache_form table each time the page is viewed. Thus, a high traffic website can have a cache_form table dozens of gigabytes big.

Another thing is that form cache in Drupal 7 is closely related to page cache and minimum cache lifetime (set on the Performance page). Clearing form cache before page cache results in the "Incorrect POST data" error returned when interacting with the form. Setting the minimum cache lifetime translates into the obsolete form cache not being cleared for the set time.

The above problem has a number of solutions.

• form_cache_expiration variable. This variable was added in version 7.61; it grants control over the form cache storage time, which is 6 hours by default. The main disadvantage of this solution is that it depends heavily on the obsolete cache clearing mechanism: if it does not fire in a timely manner, cache_form continues to grow;
• OptimizeDB module. This one allows flexible cache_form по cron. table configuring and clearing when fired by cron. The table can be cleared of old entries only or flushed of all of them. However, this solution may bring about the "Invalid POST data" error. Moreover, it does not guarantee the cache_form table will shrink to a sensible size;
• Safe cache_form Clear module. It provides a Drush command to clear obsolete entries in the cache_form table. The disadvantages of this solution are similar to those peculiar to the previous module;
• Commerce Fast Ajax Add to Cart module. This is a solution from xandeadx, it aims to solve the problem itself, i.e. fix caching of the standard Drupal Commerce Add to Cart form. Oddly enough, this is not the best choice since it does not use AJAX, and in our case we already had a coded custom add-to-cart dialogue that relied on Drupal's AJAX.. Besides, this is not a universal fix, it only works for the Add to Cart form;
• Patch #94 from this issue. In addition to applying the patch you need to add a handler for the form in question. This solution may be unstable on pages with multiple forms. It also does not work on pages showcasing a random list of products. And its major flaw is the fact that you need to patch the kernel to make it work.

Solution

Let's take a clean Drupal 7.67 instance with Commerce 1.15 module. We use Views to build the catalog pages. Each page contains 50 products; each product block includes an Add to Cart button. To speed things up, we generate products using the Commerce Devel module. The Commerce Ajax Add to Cart module helps us AJAXify the Add to cart button. Now, let's open the catalog page and check the table: yes, it gained a hundred new entries, i.e., we reproduced the problem.

The solution we suggest here is partially based on this comment. To implement it, we need to build a small custom module or add code to the existing one. In our example, we opt for a custom module.

First, let's define the path for the new AJAX form handler. Its structure is similar to "system/ajax" path in the system module.

/**
 * Implements hook_menu().
 */
function custom_menu() {
  $items['custom/form/ajax'] = array(
    'title' => 'AJAX callback',
    'page callback' => 'custom_form_ajax_callback',
    'delivery callback' => 'ajax_deliver',
    'access arguments' => array('access content'),
    'theme callback' => 'ajax_base_page_theme',
    'type' => MENU_CALLBACK,
  );
  return $items;
}

Now we need to change the Add to Cart button's AJAX handler path (path property). It is important here not to confuse the path and callback properties here: the first determines the address to which the AJAX request will be sent, the second dictates the function called in response to this request. As a rule, path is not specified and the "system/ajax", default value is taken; it needs to be changed. Also, we force disable caching of the form we are working with.

/**
 * Implements hook_form_alter().
 */
function custom_form_alter(&$form, &$form_state, $form_id) {
  if (strpos($form_id, 'commerce_cart_add_to_cart_form') !== FALSE) {
    // Указываем, что хотим самостоятельно обработать AJAX-запрос к форме.
    $form['submit']['#ajax']['path'] = 'custom/form/ajax';
 
    // Отключаем кэширование формы.
    $form_state['no_cache'] = TRUE;
  }
}

Finally, we implement the custom_form_ajax_callback() function that we previously specified in the "custom/form/ajax" path. The code of this function is partially the same to the code of ajax_get_form() and ajax_form_callback(). The idea is that we need to have the form in a correct state without cache, which we have already disabled. Important: the code below is universal, it can disable caching for other AJAX forms except for the block that forms the product, since this is the block where the form acquires the state needed for correct validation and submission. To support product attributes, we need to customize this code; other forms require the same code.

/**
 * Menu callback; handles Ajax requests for forms without caching.
 *
 * @return array|null
 *   Array of ajax commands or NULL on failure.
 */
function custom_form_ajax_callback() {
  // Проверяем, что обрабатываем AJAX-запрос к форме.
  if (isset($_POST['form_id']) && isset($_POST['form_build_id'])) {
    $form_build_id = $_POST['form_build_id'];
    $form_id = $_POST['form_id'];
    $commands = array();
 
    // Инициализируем состояние формы.
    $form_state = form_state_defaults();
    $form_state['build_info']['args'] = array();
 
    // Заполняем состояние формы. Данный код уникален в рамках обрабатываемой формы.
    // Проверяем, что форма является формой добавления товара в корзину.
    if (strpos($form_id, 'commerce_cart_add_to_cart_form_') === 0) {
      $product = commerce_product_load($_POST['product_id']);
 
      if (!empty($product)) {
        // Формируем сущность товарной позиции на основе данных отправленной формы.
        $line_item = commerce_product_line_item_new($product, $_POST['quantity'] ?? 1);
        $line_item->data['context']['product_ids'] = array($product->product_id);
        $line_item->data['context']['add_to_cart_combine'] = TRUE;
 
        // Добавляем товарную позицию в состояние формы.
        $form_state['build_info']['args'] = array($line_item);
      }
    }
 
    // Строим форму, будут вызваны билдеры и соответствующие хуки.
    $form = drupal_retrieve_form($form_id, $form_state);
    drupal_prepare_form($form_id, $form, $form_state);
    $form['#build_id_old'] = $form_build_id;
 
    // Обрабатываем форму аналогично тому, как это сделано в ajax_get_form().
    if ($form['#build_id_old'] != $form['#build_id']) {
      $commands[] = ajax_command_update_build_id($form);
    }
    $form_state['no_redirect'] = TRUE;
    $form_state['rebuild_info']['copy']['#build_id'] = TRUE;
    $form_state['rebuild_info']['copy']['#action'] = TRUE;
    $form_state['input'] = $_POST;
 
    // Обрабатываем форму аналогично тому, как это сделано в ajax_form_callback().
    drupal_process_form($form['#form_id'], $form, $form_state);
    if (!empty($form_state['triggering_element'])) {
      $callback = $form_state['triggering_element']['#ajax']['callback'];
    }
    if (!empty($callback) && is_callable($callback)) {
      $result = $callback($form, $form_state);
      if (!(is_array($result) && isset($result['#type']) && $result['#type'] == 'ajax')) {
        $result = array(
          '#type' => 'ajax',
          '#commands' => ajax_prepare_response($result),
        );
      }
      $result['#commands'] = array_merge($commands, $result['#commands']);
      return $result;
    }
  }
  return NULL;
}

Results

In the case described, adding the above code keeps the Add to Cart button AJAX-powered while the form itself is not cached anymore. Thus, refreshing the catalog page no longer yields a hundred entries in the cache_form table and the AJAX query is processed as if caching was enabled. The changes to the form that were added within Drupal API (like product attributes) either require minor code modifications (form state declaration) or none at all.

We implemented this solution on a live website, and the results are as follows:

• Ten times less SELECT queries to the cache_form table;
• Ten times less INSERT queries to the cache_form table;
• Thirty-two percent faster query processing on the server (dropped from 352 to 241 milliseconds).

See the screenshots for detailed stats.

As you can see, the average time from query to response from the APP SERVER decreased from 352 to 241 milliseconds. The size of the cache_form table shrunk from ~10GB to 200MB.

Applying the same modifications to other forms would allow improving the indicators even further.