Queues
Introduction
While building your web application, you may have some tasks, such as parsing and storing an uploaded CSV file, that take too long to perform during a typical web request. Thankfully, Laravel allows you to easily create queued jobs that may be processed in the background. By moving time intensive tasks to a queue, your application can respond to web requests with blazing speed and provide a better user experience to your customers.
Laravel queues provide a unified queueing API across a variety of different queue backends, such as Amazon SQS, Redis, or even a relational database.
Laravel's queue configuration options are stored in your application's config/queue.php
configuration file. In this file, you will find connection configurations for each of the queue drivers that are included with the framework, including the database, Amazon SQS, Redis, and Beanstalkd drivers, as well as a synchronous driver that will execute jobs immediately (for use during local development). A null
queue driver is also included which discards queued jobs.
Laravel now offers Horizon, a beautiful dashboard and configuration system for your Redis powered queues. Check out the full Horizon documentation for more information.
Connections vs. Queues
Before getting started with Laravel queues, it is important to understand the distinction between "connections" and "queues". In your config/queue.php
configuration file, there is a connections
configuration array. This option defines the connections to backend queue services such as Amazon SQS, Beanstalk, or Redis. However, any given queue connection may have multiple "queues" which may be thought of as different stacks or piles of queued jobs.
Note that each connection configuration example in the queue
configuration file contains a queue
attribute. This is the default queue that jobs will be dispatched to when they are sent to a given connection. In other words, if you dispatch a job without explicitly defining which queue it should be dispatched to, the job will be placed on the queue that is defined in the queue
attribute of the connection configuration:
use App\Jobs\ProcessPodcast;
// This job is sent to the default connection's default queue...
ProcessPodcast::dispatch();
// This job is sent to the default connection's "emails" queue...
ProcessPodcast::dispatch()->onQueue('emails');
Some applications may not need to ever push jobs onto multiple queues, instead preferring to have one simple queue. However, pushing jobs to multiple queues can be especially useful for applications that wish to prioritize or segment how jobs are processed, since the Laravel queue worker allows you to specify which queues it should process by priority. For example, if you push jobs to a high
queue, you may run a worker that gives them higher processing priority:
php artisan queue:work --queue=high,default
Driver Notes and Prerequisites
Database
In order to use the database
queue driver, you will need a database table to hold the jobs. Typically, this is included in Laravel's default 0001_01_01_000002_create_jobs_table.php
database migration; however, if your application does not contain this migration, you may use the make:queue-table
Artisan command to create it:
php artisan make:queue-table
php artisan migrate
Redis
In order to use the redis
queue driver, you should configure a Redis database connection in your config/database.php
configuration file.
The serializer
and compression
Redis options are not supported by the redis
queue driver.
Redis Cluster
If your Redis queue connection uses a Redis Cluster, your queue names must contain a key hash tag. This is required in order to ensure all of the Redis keys for a given queue are placed into the same hash slot:
'redis' => [
'driver' => 'redis',
'connection' => env('REDIS_QUEUE_CONNECTION', 'default'),
'queue' => env('REDIS_QUEUE', '{default}'),
'retry_after' => env('REDIS_QUEUE_RETRY_AFTER', 90),
'block_for' => null,
'after_commit' => false,
],
Blocking
When using the Redis queue, you may use the block_for
configuration option to specify how long the driver should wait for a job to become available before iterating through the worker loop and re-polling the Redis database.
Adjusting this value based on your queue load can be more efficient than continually polling the Redis database for new jobs. For instance, you may set the value to 5
to indicate that the driver should block for five seconds while waiting for a job to become available:
'redis' => [
'driver' => 'redis',
'connection' => env('REDIS_QUEUE_CONNECTION', 'default'),
'queue' => env('REDIS_QUEUE', 'default'),
'retry_after' => env('REDIS_QUEUE_RETRY_AFTER', 90),
'block_for' => 5,
'after_commit' => false,
],
Setting block_for
to 0
will cause queue workers to block indefinitely until a job is available. This will also prevent signals such as SIGTERM
from being handled until the next job has been processed.
Other Driver Prerequisites
The following dependencies are needed for the listed queue drivers. These dependencies may be installed via the Composer package manager:
- Amazon SQS:
aws/aws-sdk-php ~3.0
- Beanstalkd:
pda/pheanstalk ~5.0
- Redis:
predis/predis ~2.0
or phpredis PHP extension
Creating Jobs
Generating Job Classes
By default, all of the queueable jobs for your application are stored in the app/Jobs
directory. If the app/Jobs
directory doesn't exist, it will be created when you run the make:job
Artisan command:
php artisan make:job ProcessPodcast
The generated class will implement the Illuminate\Contracts\Queue\ShouldQueue
interface, indicating to Laravel that the job should be pushed onto the queue to run asynchronously.
Job stubs may be customized using stub publishing.
Class Structure
Job classes are very simple, normally containing only a handle
method that is invoked when the job is processed by the queue. To get started, let's take a look at an example job class. In this example, we'll pretend we manage a podcast publishing service and need to process the uploaded podcast files before they are published:
<?php
namespace App\Jobs;
use App\Models\Podcast;
use App\Services\AudioProcessor;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
class ProcessPodcast implements ShouldQueue
{
use Queueable;
/**
* Create a new job instance.
*/
public function __construct(
public Podcast $podcast,
) {}
/**
* Execute the job.
*/
public function handle(AudioProcessor $processor): void
{
// Process uploaded podcast...
}
}
In this example, note that we were able to pass an Eloquent model directly into the queued job's constructor. Because of the Queueable
trait that the job is using, Eloquent models and their loaded relationships will be gracefully serialized and unserialized when the job is processing.
If your queued job accepts an Eloquent model in its constructor, only the identifier for the model will be serialized onto the queue. When the job is actually handled, the queue system will automatically re-retrieve the full model instance and its loaded relationships from the database. This approach to model serialization allows for much smaller job payloads to be sent to your queue driver.
handle
Method Dependency Injection
The handle
method is invoked when the job is processed by the queue. Note that we are able to type-hint dependencies on the handle
method of the job. The Laravel service container automatically injects these dependencies.
If you would like to take total control over how the container injects dependencies into the handle
method, you may use the container's bindMethod
method. The bindMethod
method accepts a callback which receives the job and the container. Within the callback, you are free to invoke the handle
method however you wish. Typically, you should call this method from the boot
method of your App\Providers\AppServiceProvider
service provider:
use App\Jobs\ProcessPodcast;
use App\Services\AudioProcessor;
use Illuminate\Contracts\Foundation\Application;
$this->app->bindMethod([ProcessPodcast::class, 'handle'], function (ProcessPodcast $job, Application $app) {
return $job->handle($app->make(AudioProcessor::class));
});
Binary data, such as raw image contents, should be passed through the base64_encode
function before being passed to a queued job. Otherwise, the job may not properly serialize to JSON when being placed on the queue.
Queued Relationships
Because all loaded Eloquent model relationships also get serialized when a job is queued, the serialized job string can sometimes become quite large. Furthermore, when a job is deserialized and model relationships are re-retrieved from the database, they will be retrieved in their entirety. Any previous relationship constraints that were applied before the model was serialized during the job queueing process will not be applied when the job is deserialized. Therefore, if you wish to work with a subset of a given relationship, you should re-constrain that relationship within your queued job.
Or, to prevent relations from being serialized, you can call the withoutRelations
method on the model when setting a property value. This method will return an instance of the model without its loaded relationships:
/**
* Create a new job instance.
*/
public function __construct(Podcast $podcast)
{
$this->podcast = $podcast->withoutRelations();
}
If you are using PHP constructor property promotion and would like to indicate that an Eloquent model should not have its relations serialized, you may use the WithoutRelations
attribute:
use Illuminate\Queue\Attributes\WithoutRelations;
/**
* Create a new job instance.
*/
public function __construct(
#[WithoutRelations]
public Podcast $podcast
) {
}
If a job receives a collection or array of Eloquent models instead of a single model, the models within that collection will not have their relationships restored when the job is deserialized and executed. This is to prevent excessive resource usage on jobs that deal with large numbers of models.
Unique Jobs
Unique jobs require a cache driver that supports locks. Currently, the memcached
, redis
, dynamodb
, database
, file
, and array
cache drivers support atomic locks. In addition, unique job constraints do not apply to jobs within batches.
Sometimes, you may want to ensure that only one instance of a specific job is on the queue at any point in time. You may do so by implementing the ShouldBeUnique
interface on your job class. This interface does not require you to define any additional methods on your class:
<?php
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Contracts\Queue\ShouldBeUnique;
class UpdateSearchIndex implements ShouldQueue, ShouldBeUnique
{
...
}
In the example above, the UpdateSearchIndex
job is unique. So, the job will not be dispatched if another instance of the job is already on the queue and has not finished processing.
In certain cases, you may want to define a specific "key" that makes the job unique or you may want to specify a timeout beyond which the job no longer stays unique. To accomplish this, you may define uniqueId
and uniqueFor
properties or methods on your job class:
<?php
use App\Models\Product;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Contracts\Queue\ShouldBeUnique;
class UpdateSearchIndex implements ShouldQueue, ShouldBeUnique
{
/**
* The product instance.
*
* @var \App\Product
*/
public $product;
/**
* The number of seconds after which the job's unique lock will be released.
*
* @var int
*/
public $uniqueFor = 3600;
/**
* Get the unique ID for the job.
*/
public function uniqueId(): string
{
return $this->product->id;
}
}
In the example above, the UpdateSearchIndex
job is unique by a product ID. So, any new dispatches of the job with the same product ID will be ignored until the existing job has completed processing. In addition, if the existing job is not processed within one hour, the unique lock will be released and another job with the same unique key can be dispatched to the queue.
If your application dispatches jobs from multiple web servers or containers, you should ensure that all of your servers are communicating with the same central cache server so that Laravel can accurately determine if a job is unique.
Keeping Jobs Unique Until Processing Begins
By default, unique jobs are "unlocked" after a job completes processing or fails all of its retry attempts. However, there may be situations where you would like your job to unlock immediately before it is processed. To accomplish this, your job should implement the ShouldBeUniqueUntilProcessing
contract instead of the ShouldBeUnique
contract:
<?php
use App\Models\Product;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Contracts\Queue\ShouldBeUniqueUntilProcessing;
class UpdateSearchIndex implements ShouldQueue, ShouldBeUniqueUntilProcessing
{
// ...
}
Unique Job Locks
Behind the scenes, when a ShouldBeUnique
job is dispatched, Laravel attempts to acquire a lock with the uniqueId
key. If the lock is not acquired, the job is not dispatched. This lock is released when the job completes processing or fails all of its retry attempts. By default, Laravel will use the default cache driver to obtain this lock. However, if you wish to use another driver for acquiring the lock, you may define a uniqueVia
method that returns the cache driver that should be used:
use Illuminate\Contracts\Cache\Repository;
use Illuminate\Support\Facades\Cache;
class UpdateSearchIndex implements ShouldQueue, ShouldBeUnique
{
...
/**
* Get the cache driver for the unique job lock.
*/
public function uniqueVia(): Repository
{
return Cache::driver('redis');
}
}
If you only need to limit the concurrent processing of a job, use the WithoutOverlapping
job middleware instead.
Encrypted Jobs
Laravel allows you to ensure the privacy and integrity of a job's data via encryption. To get started, simply add the ShouldBeEncrypted
interface to the job class. Once this interface has been added to the class, Laravel will automatically encrypt your job before pushing it onto a queue:
<?php
use Illuminate\Contracts\Queue\ShouldBeEncrypted;
use Illuminate\Contracts\Queue\ShouldQueue;
class UpdateSearchIndex implements ShouldQueue, ShouldBeEncrypted
{
// ...
}
Job Middleware
Job middleware allow you to wrap custom logic around the execution of queued jobs, reducing boilerplate in the jobs themselves. For example, consider the following handle
method which leverages Laravel's Redis rate limiting features to allow only one job to process every five seconds:
use Illuminate\Support\Facades\Redis;
/**
* Execute the job.
*/
public function handle(): void
{
Redis::throttle('key')->block(0)->allow(1)->every(5)->then(function () {
info('Lock obtained...');
// Handle job...
}, function () {
// Could not obtain lock...
return $this->release(5);
});
}
While this code is valid, the implementation of the handle
method becomes noisy since it is cluttered with Redis rate limiting logic. In addition, this rate limiting logic must be duplicated for any other jobs that we want to rate limit.
Instead of rate limiting in the handle method, we could define a job middleware that handles rate limiting. Laravel does not have a default location for job middleware, so you are welcome to place job middleware anywhere in your application. In this example, we will place the middleware in an app/Jobs/Middleware
directory:
<?php
namespace App\Jobs\Middleware;
use Closure;
use Illuminate\Support\Facades\Redis;
class RateLimited
{
/**
* Process the queued job.
*
* @param \Closure(object): void $next
*/
public function handle(object $job, Closure $next): void
{
Redis::throttle('key')
->block(0)->allow(1)->every(5)
->then(function () use ($job, $next) {
// Lock obtained...
$next($job);
}, function () use ($job) {
// Could not obtain lock...
$job->release(5);
});
}
}
As you can see, like route middleware, job middleware receive the job being processed and a callback that should be invoked to continue processing the job.
After creating job middleware, they may be attached to a job by returning them from the job's middleware
method. This method does not exist on jobs scaffolded by the make:job
Artisan command, so you will need to manually add it to your job class:
use App\Jobs\Middleware\RateLimited;
/**
* Get the middleware the job should pass through.
*
* @return array<int, object>
*/
public function middleware(): array
{
return [new RateLimited];
}
Job middleware can also be assigned to queueable event listeners, mailables, and notifications.
Rate Limiting
Although we just demonstrated how to write your own rate limiting job middleware, Laravel actually includes a rate limiting middleware that you may utilize to rate limit jobs. Like route rate limiters, job rate limiters are defined using the RateLimiter
facade's for
method.
For example, you may wish to allow users to backup their data once per hour while imposing no such limit on premium customers. To accomplish this, you may define a RateLimiter
in the boot
method of your AppServiceProvider
:
use Illuminate\Cache\RateLimiting\Limit;
use Illuminate\Support\Facades\RateLimiter;
/**
* Bootstrap any application services.
*/
public function boot(): void
{
RateLimiter::for('backups', function (object $job) {
return $job->user->vipCustomer()
? Limit::none()
: Limit::perHour(1)->by($job->user->id);
});
}
In the example above, we defined an hourly rate limit; however, you may easily define a rate limit based on minutes using the perMinute
method. In addition, you may pass any value you wish to the by
method of the rate limit; however, this value is most often used to segment rate limits by customer:
return Limit::perMinute(50)->by($job->user->id);
Once you have defined your rate limit, you may attach the rate limiter to your job using the Illuminate\Queue\Middleware\RateLimited
middleware. Each time the job exceeds the rate limit, this middleware will release the job back to the queue with an appropriate delay based on the rate limit duration.
use Illuminate\Queue\Middleware\RateLimited;
/**
* Get the middleware the job should pass through.
*
* @return array<int, object>
*/
public function middleware(): array
{
return [new RateLimited('backups')];
}
Releasing a rate limited job back onto the queue will still increment the job's total number of attempts
. You may wish to tune your tries
and maxExceptions
properties on your job class accordingly. Or, you may wish to use the retryUntil
method to define the amount of time until the job should no longer be attempted.
If you do not want a job to be retried when it is rate limited, you may use the dontRelease
method:
/**
* Get the middleware the job should pass through.
*
* @return array<int, object>
*/
public function middleware(): array
{
return [(new RateLimited('backups'))->dontRelease()];
}
If you are using Redis, you may use the Illuminate\Queue\Middleware\RateLimitedWithRedis
middleware, which is fine-tuned for Redis and more efficient than the basic rate limiting middleware.
Preventing Job Overlaps
Laravel includes an Illuminate\Queue\Middleware\WithoutOverlapping
middleware that allows you to prevent job overlaps based on an arbitrary key. This can be helpful when a queued job is modifying a resource that should only be modified by one job at a time.
For example, let's imagine you have a queued job that updates a user's credit score and you want to prevent credit score update job overlaps for the same user ID. To accomplish this, you can return the WithoutOverlapping
middleware from your job's middleware
method:
use Illuminate\Queue\Middleware\WithoutOverlapping;
/**
* Get the middleware the job should pass through.
*
* @return array<int, object>
*/
public function middleware(): array
{
return [new WithoutOverlapping($this->user->id)];
}
Any overlapping jobs of the same type will be released back to the queue. You may also specify the number of seconds that must elapse before the released job will be attempted again:
/**
* Get the middleware the job should pass through.
*
* @return array<int, object>
*/
public function middleware(): array
{
return [(new WithoutOverlapping($this->order->id))->releaseAfter(60)];
}
If you wish to immediately delete any overlapping jobs so that they will not be retried, you may use the dontRelease
method:
/**
* Get the middleware the job should pass through.
*
* @return array<int, object>
*/
public function middleware(): array
{
return [(new WithoutOverlapping($this->order->id))->dontRelease()];
}
The WithoutOverlapping
middleware is powered by Laravel's atomic lock feature. Sometimes, your job may unexpectedly fail or timeout in such a way that the lock is not released. Therefore, you may explicitly define a lock expiration time using the expireAfter
method. For example, the example below will instruct Laravel to release the WithoutOverlapping
lock three minutes after the job has started processing:
/**
* Get the middleware the job should pass through.
*
* @return array<int, object>
*/
public function middleware(): array
{
return [(new WithoutOverlapping($this->order->id))->expireAfter(180)];
}
The WithoutOverlapping
middleware requires a cache driver that supports locks. Currently, the memcached
, redis
, dynamodb
, database
, file
, and array
cache drivers support atomic locks.
Sharing Lock Keys Across Job Classes
By default, the WithoutOverlapping
middleware will only prevent overlapping jobs of the same class. So, although two different job classes may use the same lock key, they will not be prevented from overlapping. However, you can instruct Laravel to apply the key across job classes using the shared
method:
use Illuminate\Queue\Middleware\WithoutOverlapping;
class ProviderIsDown
{
// ...
public function middleware(): array
{
return [
(new WithoutOverlapping("status:{$this->provider}"))->shared(),
];
}
}
class ProviderIsUp
{
// ...
public function middleware(): array
{
return [
(new WithoutOverlapping("status:{$this->provider}"))->shared(),
];
}
}
Throttling Exceptions
Laravel includes a Illuminate\Queue\Middleware\ThrottlesExceptions
middleware that allows you to throttle exceptions. Once the job throws a given number of exceptions, all further attempts to execute the job are delayed until a specified time interval lapses. This middleware is particularly useful for jobs that interact with third-party services that are unstable.
For example, let's imagine a queued job that interacts with a third-party API that begins throwing exceptions. To throttle exceptions, you can return the ThrottlesExceptions
middleware from your job's middleware
method. Typically, this middleware should be paired with a job that implements time based attempts:
use DateTime;
use Illuminate\Queue\Middleware\ThrottlesExceptions;
/**
* Get the middleware the job should pass through.
*
* @return array<int, object>
*/
public function middleware(): array
{
return [new ThrottlesExceptions(10, 5)];
}
/**
* Determine the time at which the job should timeout.
*/
public function retryUntil(): DateTime
{
return now()->addMinutes(5);
}
The first constructor argument accepted by the middleware is the number of exceptions the job can throw before being throttled, while the second constructor argument is the number of minutes that should elapse before the job is attempted again once it has been throttled. In the code example above, if the job throws 10 exceptions within 5 minutes, we will wait 5 minutes before attempting the job again.
When a job throws an exception but the exception threshold has not yet been reached, the job will typically be retried immediately. However, you may specify the number of minutes such a job should be delayed by calling the backoff
method when attaching the middleware to the job:
use Illuminate\Queue\Middleware\ThrottlesExceptions;
/**
* Get the middleware the job should pass through.
*
* @return array<int, object>
*/
public function middleware(): array
{
return [(new ThrottlesExceptions(10, 5))->backoff(5)];
}
Internally, this middleware uses Laravel's cache system to implement rate limiting, and the job's class name is utilized as the cache "key". You may override this key by calling the by
method when attaching the middleware to your job. This may be useful if you have multiple jobs interacting with the same third-party service and you would like them to share a common throttling "bucket":
use Illuminate\Queue\Middleware\ThrottlesExceptions;
/**
* Get the middleware the job should pass through.
*
* @return array<int, object>
*/
public function middleware(): array
{
return [(new ThrottlesExceptions(10, 10))->by('key')];
}
By default, this middleware will throttle every exception. You can modify this behaviour by invoking the when
method when attaching the middleware to your job. The exception will then only be throttled if closure provided to the when
method returns true
:
use Illuminate\Http\Client\HttpClientException;
use Illuminate\Queue\Middleware\ThrottlesExceptions;
/**
* Get the middleware the job should pass through.
*
* @return array<int, object>
*/
public function middleware(): array
{
return [(new ThrottlesExceptions(10, 10))->when(
fn (Throwable $throwable) => $throwable instanceof HttpClientException
)];
}
If you would like to have the throttled exceptions reported to your application's exception handler, you can do so by invoking the report
method when attaching the middleware to your job. Optionally, you may provide a closure to the report
method and the exception will only be reported if the given closure returns true
:
use Illuminate\Http\Client\HttpClientException;
use Illuminate\Queue\Middleware\ThrottlesExceptions;
/**
* Get the middleware the job should pass through.
*
* @return array<int, object>
*/
public function middleware(): array
{
return [(new ThrottlesExceptions(10, 10))->report(
fn (Throwable $throwable) => $throwable instanceof HttpClientException
)];
}
If you are using Redis, you may use the Illuminate\Queue\Middleware\ThrottlesExceptionsWithRedis
middleware, which is fine-tuned for Redis and more efficient than the basic exception throttling middleware.
Dispatching Jobs
Once you have written your job class, you may dispatch it using the dispatch
method on the job itself. The arguments passed to the dispatch
method will be given to the job's constructor:
<?php
namespace App\Http\Controllers;
use App\Http\Controllers\Controller;
use App\Jobs\ProcessPodcast;
use App\Models\Podcast;
use Illuminate\Http\RedirectResponse;
use Illuminate\Http\Request;
class PodcastController extends Controller
{
/**
* Store a new podcast.
*/
public function store(Request $request): RedirectResponse
{
$podcast = Podcast::create(/* ... */);
// ...
ProcessPodcast::dispatch($podcast);
return redirect('/podcasts');
}
}
If you would like to conditionally dispatch a job, you may use the dispatchIf
and dispatchUnless
methods:
ProcessPodcast::dispatchIf($accountActive, $podcast);
ProcessPodcast::dispatchUnless($accountSuspended, $podcast);
In new Laravel applications, the sync
driver is the default queue driver. This driver executes jobs synchronously in the foreground of the current request, which is often convenient during local development. If you would like to actually begin queueing jobs for background processing, you may specify a different queue driver within your application's config/queue.php
configuration file.
Delayed Dispatching
If you would like to specify that a job should not be immediately available for processing by a queue worker, you may use the delay
method when dispatching the job. For example, let's specify that a job should not be available for processing until 10 minutes after it has been dispatched:
<?php
namespace App\Http\Controllers;
use App\Http\Controllers\Controller;
use App\Jobs\ProcessPodcast;
use App\Models\Podcast;
use Illuminate\Http\RedirectResponse;
use Illuminate\Http\Request;
class PodcastController extends Controller
{
/**
* Store a new podcast.
*/
public function store(Request $request): RedirectResponse
{
$podcast = Podcast::create(/* ... */);
// ...
ProcessPodcast::dispatch($podcast)
->delay(now()->addMinutes(10));
return redirect('/podcasts');
}
}
The Amazon SQS queue service has a maximum delay time of 15 minutes.
Dispatching After the Response is Sent to the Browser
Alternatively, the dispatchAfterResponse
method delays dispatching a job until after the HTTP response is sent to the user's browser if your web server is using FastCGI. This will still allow the user to begin using the application even though a queued job is still executing. This should typically only be used for jobs that take about a second, such as sending an email. Since they are processed within the current HTTP request, jobs dispatched in this fashion do not require a queue worker to be running in order for them to be processed:
use App\Jobs\SendNotification;
SendNotification::dispatchAfterResponse();