Getting Started With Laravel Queues and Background Jobs

This tutorial will show you how to configure Laravel to use a database for scheduling/queuing jobs, although you can use the majority of the information to help set up a different queuing driver.

If you don't do this, and you are using the stock driver, then all jobs are executed immediately, instead of asynchronously, which will impact the performance of your system. E.g. you may think you are creating a job that will run later in the background, but actually it isn't.

Steps

First we need to create our class for a background job. In this case I am creating a background job for sending a welcome email to a user.

php artisan make:job EmailUserWelcomeEmail

You will now see the class within /app/Jobs

The class will implement the ShouldQueue interface.

To prevent issues with loading, put parameters in constructor and pass parameters using ::dispatch

\App\Jobs\JobSendUserWelcomeEmail::dispatch($newUser->getId());

Configure Laravel To Use Database Driver

When I first ran this, I saw that the code was not running these jobs in the background. It turns out this was because my config/queue.php file had the queue driver set to sync. This essentially means there is no queue and the jobs run immdediately. (More info).

The simplest solution to get an asynchronous queue working was to switch to the database driver. Before enabling this driver, we need to make sure the queue table is created as well as a table to hold details of any failed jobs:

php artisan queue:table
php artisan queue:failed-table

This creates the migration script to create the "jobs" table like so:

Schema::create('jobs', function (Blueprint $table) {
    $table->bigIncrements('id');
    $table->string('queue')->index();
    $table->longText('payload');
    $table->unsignedTinyInteger('attempts');
    $table->unsignedInteger('reserved_at')->nullable();
    $table->unsignedInteger('available_at');
    $table->unsignedInteger('created_at');
});

... and the failed jobs table like so:

Schema::create('failed_jobs', function (Blueprint $table) {
    $table->bigIncrements('id');
    $table->text('connection');
    $table->text('queue');
    $table->longText('payload');
    $table->longText('exception');
    $table->timestamp('failed_at')->useCurrent();
});

Using PostgreSQL With UUIDs for IDs

If you want to change these to use UUID IDs if you are using postgreSQL so change to be like so:

Schema::create('jobs', function (Blueprint $table) {
    $table->uuid('id')->primary(); // this line changed
    $table->string('queue')->index();
    $table->longText('payload');
    $table->unsignedTinyInteger('attempts');
    $table->unsignedInteger('reserved_at')->nullable();
    $table->unsignedInteger('available_at');
    $table->unsignedInteger('created_at');
});

Schema::create('failed_jobs', function (Blueprint $table) {
    $table->uuid('id')->primary(); // this line changed
    $table->text('connection');
    $table->text('queue');
    $table->longText('payload');
    $table->longText('exception');
    $table->timestamp('failed_at')->useCurrent();
});

You would also need to update the failed jobs table to specify that uses database-uuids driver:

    /*
    |--------------------------------------------------------------------------
    | Failed Queue Jobs
    |--------------------------------------------------------------------------
    |
    | These options configure the behavior of failed queue job logging so you
    | can control which database and table are used to store the jobs that
    | have failed. You may change them to any database / table you wish.
    |
    */

    'failed' => [
        'driver' => 'database-uuids',
        'database' => 'pgsql',
        'table' => 'laravel_failed_jobs',
    ],

Run Migrations

Then you need to run migrations to run this new migration:

php artisan migrate

Specify the Database Driver

Now you can go to /config/queue.php and update the queue driver like so:

    /*
    |--------------------------------------------------------------------------
    | Default Queue Connection Name
    |--------------------------------------------------------------------------
    |
    | Laravel's queue API supports an assortment of back-ends via a single
    | API, giving you convenient access to each back-end using the same
    | syntax for every one. Here you may define a default connection.
    |
    */

    'default' => env('QUEUE_CONNECTION', 'database'),

Also, make sure that if QUEUE_CONNECTION is set in your .env file, that you update it to state database.

Older versions of Laravel referred to the environment variable as QUEUE_DRIVER instead. Refer here.

Configure Supervisor

After performing all of the steps above, I saw that my request was returning immediately, but I wasn't getting any emails. Looking in the jobs table, I could see the jobs there with an attempts count set to 0. It appears that nothing was actually running the queue. To do this, the easiest thing to do is configure Supervisor, by adding the following configuration to the Supervisor config file:

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/my-site/artisan queue:work database --sleep=3 --tries=3
autostart=true
autorestart=true
user=www-data
numprocs=1
redirect_stderr=true
stdout_logfile=/var/www/my-site/storage/logs/worker.log

In this example, the numprocs directive will instruct Supervisor to run just one queue:work processes and monitor it, automatically restarting it if it fails. Logs of errors will be redirected to the log file within laravel's storage directory.

Testing / Commands

Once you have hooked all of this up, you probably want to quickly test that it is all working which we can do by manually scheduling work and running the worker that processes the tasks.

If you are using my Github Laravel template, then it is already running a worker in the background through the supervisor configuration as outlined above. It is also running the scheduler every minute through the cron configuration, in order to have the scheduler schedule jobs. For testing/dev purposes, you may wish to disable both of these by running supervisor stop, and commenting out the cron job. Then you can just make use of the commands below:

Schedule:Work

You can manually tell the scheduler to run once, and schedule tasks by calling:

php artisan schedule:run

Any jobs that are scheduled to run every minute will essentially be scheduled every time you call this command. This is because the scheduler is not actually time-based, but relies on the command being called once a minute.

In newer versions of Laravel, if you have tasks scheduled to run several times in a minute, what actually happens is that the scheduler keeps running for the entire minute that it was called, and will schedule the jobs at the appropriate intervals in that time. Please refer here for the official docs where they explain this.

Schedule:Run

If you do not wish to be manually calling php artisan schedule:run, then you can run:

php artisan schedule:work

This will stay running forever, and essentially calls php artisan schedule:run once per minute for you.

Run The Worker - Queue:Work

Scheduling jobs without any workers running gives you the opportunity to see the jobs queing up in the database in their serialized form.

Once you are ready for the jobs to actually be executed, you can execute them by running:

php artisan queue:work database

This will try to execute all of the queued up jobs one at a time. If the tasks take a while, you could run multiple workers by running the command in other terminals.

References

• Stack Overflow - Passing parameters to Laravel job is not working
• Laravel Docs- Supervisor Configuration
• Laravel Docs - Dealing With Failed Jobs
• riptutorial.com - Queue Driver Configuration
• Laracasts - Queue It Up - Episode 1
• Laravel Docs - Generating Job Classes