Knowledgebase

Introduction

Redis queue is a library that you can use to create and process a linear collection of jobs on a first-in-first-out basis. You should always use queues in your web applications to avoid keeping end-users waiting for a response if your business logic takes a long time to complete. For instance, in a busy online store, you can perform complex processes like payment and address verification using queues and background workers. In other words, if you have a use-case where you want processes to occur in an orderly manner but your web application can't keep up with the end user's patience, consider using a Redis queue, such as a file conversion portal.

In this guide, you'll use the Redis RPUSH and LPOP commands with PHP and MySQL to process customers' registrations in a MySQL database on Ubuntu 20.04 server.

Prerequisites

To complete the tutorial, you'll need the following:

• An Ubuntu 20.04 server.
• A sudo user.
• A LAMP Stack.
• A Redis Server.

Install php-redis Extension

Connect to your server, update the package information index and install php-redis helper library that allows you to use the Redis server's functionalities inside your PHP code.

$ sudo apt update
$ sudo apt install -y php-redis

Restart the Apache webserver to reload the changes.

$ sudo systemctl restart apache2

Create a test_db Database

In this tutorial, you will queue the customers' registration information in the Redis server, but later, you'll persistently save the data in the MySQL database.

Run the command below to log in to the MySQL server as a root user.

$ sudo mysql -u root -p

When prompted, key in the root password of your MySQL server and press Enter to proceed. Then, create a test_db database.

mysql> CREATE DATABASE test_db;

Set up a test_db_user user and use a strong value for the EXAMPLE_PASSWORD. You will use these details to connect to MySQL from your PHP script.

mysql> CREATE USER 'test_db_user'@'localhost' IDENTIFIED WITH mysql_native_password BY 'EXAMPLE_PASSWORD';
mysql> GRANT ALL PRIVILEGES ON test_db.* TO 'test_db_user'@'localhost';
mysql> FLUSH PRIVILEGES;

Change the CREATE USER command to the below syntax if you're using the MariaDB server.

MariaDB> GRANT ALL PRIVILEGES on test_db.* TO 'test_db_user'@'localhost' identified by 'EXAMPLE_PASSWORD';

Then, switch to the test_db database.

mysql> USE test_db;

Create a customers table by running the command below.

mysql> CREATE TABLE customers (
       customer_id BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY,
       first_name VARCHAR(50),
       last_name VARCHAR(50),
       email_address VARCHAR(255)
       ) ENGINE = InnoDB;

Don't enter any data into the customers table, you'll later populate the table using a job that fetches data from the Redis server. Exit from the mysql> command-line interface.

mysql> QUIT;

Set Up a Redis Queue PHP File

Open a new /var/www/html/redis_queue.php using nano in the root directory of your web server. This file will receive POST parameters with customer details for registration purposes.

$ sudo nano /var/www/html/redis_queue.php

Add the information below to the file.

<?php
    try {
            $redis = new Redis();
            $redis->connect('127.0.0.1', 6379);

            $data = [];
            $data = [
                    'first_name'    => $_POST['first_name'],
                    'last_name'     => $_POST['last_name'],
                    'email_address' => $_POST['email_address']
                    ];

            $redis->rpush("customers_register", json_encode($data));

            echo "Customer details accepted successfully.\n";

        } catch (Exception $e) {
            echo $e->getMessage();
        }

Save and close the file.

In the file above, you're connecting to the locally hosted Redis server at port 6379. Then, you're creating a $data array of the customer's data as received from the global $_POST parameter. Then, you're pushing the data as a JSON encoded string to the Redis server using the $redis->rpush command.

Once the data is saved to the Redis server, you're echoing out a success message. In the next step, you'll send sample data to the file.

Send Customer's Data to the Redis Queue

In a production environment, customers may subscribe to your application using HTML forms. For the basis of this tutorial, you'll just use the Linux curl command to send the data to the http://localhost/queue_register.php that you've created above.

Send three customers' records by running the curl commands below one by one.

$ curl --data "first_name=JOHN&last_name=DOE&email_address=john_doe@example.com" http://localhost/redis_queue.php
$ curl --data "first_name=MARY&last_name=SMITH&email_address=mary_smith@example.com" http://localhost/redis_queue.php
$ curl --data "first_name=ROE&last_name=STEVE&email_address=roe_steve@example.com" http://localhost/redis_queue.php

Ensure you get the following output after issuing each command.

...

Customer details accepted successfully.

The output above confirms that the Redis server is queueing data successfully. You'll now move on to creating a worker for processing the queue.

Save the Redis Data Permanently to MySQL

In this step, you'll use the Redis LPOP command to dequeue the customers' data from the Redis server and permanently save the data to the MySQL database.

Open a new /var/www/html/redis_worker.php file using nano.

$ sudo nano /var/www/html/redis_worker.php

Then, add the information below into the file.

<?php
    try {
            $redis = new Redis();
            $redis->connect('127.0.0.1', 6379);

            $data = $redis->lpop('customers_register');
            $data  = json_decode($data, true);

            if (!empty($data)) {
               $db_name     = 'test_db';
               $db_user     = 'test_db_user';
               $db_password = 'EXAMPLE_PASSWORD';
               $db_host     = 'localhost';

               $pdo = new PDO('mysql:host=' . $db_host . '; dbname=' . $db_name, $db_user, $db_password);
               $pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
               $pdo->setAttribute(PDO::ATTR_EMULATE_PREPARES, false);

               $sql = 'insert into customers
                      (
                      first_name,
                      last_name,
                      email_address
                      )
                      values
                      (
                      :first_name,
                      :last_name,
                      :email_address
                      )
                      ';

               $stmt = $pdo->prepare($sql);
               $stmt->execute($data);

               echo $data[first_name] . " " . $data[last_name] . "'s details saved to database.";

            }

        } catch (Exception $e) {
            echo $e->getMessage();
        }

Save and close the file.

In the above file, you're connecting to the Redis server and retrieving customers' data on a first-in-first-out basis using $redis->lpop statement. Next, you've JSON decoded the data and constructed an SQL command to be executed by the PHP Data Object($pdo). Finally, you're saving the data permanently to the MySQL server and echoing out a success message.

Please note, you're using bound parameters to prevent SQL injection attacks. This guide is for demonstration purposes and does not include any form of data validation. In a production environment, you would sanitize and filter the data even further to have a solid security layer.

Testing the Redis Server Worker

Execute the http://localhost/redis_worker.php URL to test if the redis_worker.php file is working as expected. Run the command below three times to process all the entries.

$ curl http://localhost/redis_worker.php

As you can see from the output below, the Redis LPOP command is successfully registering the customers in the database in the order their data arrived in the Redis server.

JOHN DOE's details saved to database.
MARY SMITH's details saved to database.
ROE STEVE's details saved to database.

Confirm the customers' details from the database to make sure they are in place. To do this, log back to the MySQL server as root.

$ sudo mysql -u root -p

Enter your password and press Enter to continue. Then, switch to the test_db database.

mysql> USE test_db;

Run a SELECT statement against the customers table.

mysql> SELECT
       customer_id,
       first_name,
       last_name,
       email_address
       FROM customers;

You should see the output below to confirm that the Redis queue's and worker's logic are working as expected.

+-------------+------------+-----------+------------------------+
| customer_id | first_name | last_name | email_address          |
+-------------+------------+-----------+------------------------+
|           1 | JOHN       | DOE       | john_doe@example.com   |
|           2 | MARY       | SMITH     | mary_smith@example.com |
|           3 | ROE        | STEVE     | roe_steve@example.com  |
+-------------+------------+-----------+------------------------+
3 rows in set (0.00 sec)

Exit from the MySQL command-line interface

mysql> QUIT;

Run the Redis Worker Automatically

In the previous step, you've run the http://localhost/redis_worker.php script manually using the Linux curl command. To automatically run the worker, you need to create a cron job that executes the file without your action.

Open the crontab file.

$ sudo nano /etc/crontab

Add the content below to the bottom of the file to run the http://localhost/redis_worker.php worker every minute.

...
* * * * * root /usr/bin/wget -O - http://localhost/redis_worker.php

Save and close the file. Now, send another customer's data to the http://localhost/redis_queue.php endpoint.

$ curl --data "first_name=BABY&last_name=SMALL&email_address=baby_small@example.com" http://localhost/redis_queue.php

Make sure the customer's details are queued.

Customer details accepted successfully.

Log back to the MySQL server after one minute.

$ sudo mysql -u root -p

Key in the root password of your MySQL server and press Enter. Then, use the test_db database.

mysql> USE test_db;

Execute the SELECT statement below one more time to see if the cron job is executing the worker.

mysql> SELECT
       customer_id,
       first_name,
       last_name,
       email_address
       FROM customers;

You should get the out below confirming that the cron job is working as expected.

+-------------+------------+-----------+------------------------+
| customer_id | first_name | last_name | email_address          |
+-------------+------------+-----------+------------------------+
|           1 | JOHN       | DOE       | john_doe@example.com   |
|           2 | MARY       | SMITH     | mary_smith@example.com |
|           3 | ROE        | STEVE     | roe_steve@example.com  |
|           4 | BABY       | SMALL     | baby_small@example.com |
+-------------+------------+-----------+------------------------+
4 rows in set (0.00 sec)

A one-minute time-period may be too long for some mission-critical applications, and in such cases, you can create a PHP loop file that keeps polling the worker file for new queue entries.

Conclusion

You've used the Redis server to queue and process jobs with PHP on Ubuntu 20.04 server in this guide. You may extend the code in this guide to suit your needs.