The great thing about the server architectures we have these days is that everything is scalable (if you set it up correctly). The hard thing about the server architectures we have these days is that everything is scalable (if you set it up correctly). Yeah, I know.
One of the things you may run into these days (and that I had to solve this week) is that these days we provision all servers similarly (or according to their role). This may also mean that you provision several of your servers to run the same cronjobs at the same time. However, some tasks may not be run multiple times, and especially not at the same time. In a symfony project I’m working on, I was tasked with making sure some of the cronjobs would only be run once, even if started on several servers at the same time.
Adding the locking
My initial idea was to add a locking system to all commands that had to be idempotent, but I felt this was a bad idea: Having to add similar code to several different classes did not really make sense to me.
While looking for a different option by searching for Symfony Command classes and events I came by this blogpost by Matthias Noback. While his specific use case in that blogpost is different, it inspired me: I simply needed to use the events console.command
and console.terminate
. I would be able to hook into those events to initially set the lock and then on termination release the lock.
The lock library
The next step was to find the right type of locking. I looked around for libraries that could do locking. In that process I came by the very recently pushed symfony/lock. Unfortunately that was a bit too fresh for me to use. Eventually, I settled on arvenil/ninja-mutex, a nice and simple library that can do Mutex locks on a variety of backends. We went with the Redis backend for our locks.
Deciding when to lock
The thing is: I don’t need all commands to do locking, I only need specific commands to add and release locks. My initial plan was to simply create an array of class names in the listener, but that did not feel right. This meant that every time we’d add a new command that needs locking, we’d have to update the listener.
Another option would be to keep track of a list in the configuration, but that similarly did not feel right.
I ended up going for an implementation with an interface. The interface, that I called IdempotentCommand
, contains just a single method. The method that needs to be implemented is getIdentifier(): string
, which would return the identifier used for the lock.
The listener
Time to write the listener. The listener needs to listen to two events:
console.command
is the event triggered by starting a Symfony Command. This is where I need to create the lock.console.terminate
is the event triggered by a Symfony Command ending execution. This is where I need to release the lock.
The listener is pretty simple. It gets the MutexFabric
class from arvenil/ninja-mutex
as a constructor argument that it can use internally. It then implements two methods, one for the first event and one from the second event.
Creating the lock
public function onConsoleCommand(ConsoleCommandEvent $event)
{
if ($this->shouldBeHandledIdempotently($event->getCommand())) {
$this->acquireLock($this->getLockName($event->getCommand()->getIdentifier()));
}
}
private function shouldBeHandledIdempotently(Command $command)
{
return $command instanceof IdempotentCommand;
}
private function acquireLock(string $name)
{
$result = $this->mutexPool->get($name)->acquireLock(1000);
if (false === $result) {
throw new ProcessLocked('Process '.$name.' is locked and can not be executed');
}
$this->acquired = true;
}
private function getLockName(string $commandName): string
{
return 'command-'.$commandName;
}
The onConsoleCommand()
method is linked to the console.command
using a service tag:
- { name: kernel.event_listener, event: console.command, method: onConsoleCommand, priority: 1 }
First we check whether this command is required to be locked. If so, we try to acquire a lock. If we succeed, we keep track of that by setting a local property (yay, we are the actually executing process). The purpose of this property is to prevent a second (or third) process that is started to release the lock when it ends before the initial process is ended. If we can not acquire a lock we throw a ProcessLocked exception to quit execution immediately.
Releasing the lock
Once the main process has ended, it needs to release the lock. To do that, we have a second method in our listener class:
public function onConsoleTerminate(ConsoleTerminateEvent $event)
{
if ($this->shouldBeHandledIdempotently($event->getCommand()) && $this->acquired === true) {
$this->releaseLock($this->getLockName($event->getCommand()->getIdentifier()));
}
}
private function releaseLock(string $name)
{
$this->mutexPool->get($name)->releaseLock();
}
Here we simply check whether this command is supposed to lock and whether the current process has acquired the lock. If so, it releases the lock. That’s all.
Making things lock
Now the only step left is to find the right Command classes that need to be locked, and make them implement the IdempotentCommand
interface I defined at the start. These are now automatically picked up by the listener to set and release a lock accordingly.
If I now start the same command twice at the same time, only one of the commands will actually run, the other one will be stopped by the exception and won’t run at all.
A small extra lesson
During the process of building the second listener method, some weird things were happening. I would get an error at the end of the Command execution about the locks, and whatever I did it seemed the second listener was never triggered. After a lot of searching I found the error to be a single missing comma. I had accidentally typed:
- { name: kernel.event_listener, event: console.terminate method: onConsoleTerminate, priority: 1 }
This is still valid YAML, so Symfony did not complain about it, but because of the missing comma between console.terminate
and method:
it did not pick it up to be a listener. The devil is in the details, and it took me a while to figure this one out.
Leave a Reply