Distributed application in PHP with Apache Zookeeper

Apache ZooKeeper is the coolest technology I recently came across. I found it when I was doing a research about Solr Cloud features. I got very impressed by Solr’s distributed computing. You literately have to fire a new instance and it will automatically find its place in “the cloud”. It will assign itself to a particular shards and it will make a decision to become a leader or a replica. Later you can query any of the available servers and it will find you all required data even if it’s not on that server. If some of the servers fail the service will continue to work. Very dynamic, very clever, very cool.

Running multiple application as one logical program is nothing new. In fact creating such a software was one of my first jobs many years ago. This type of architecture is confusing and very tricky to work with. Apache ZooKeeper tries to provide a generic set of tools to manage such a software.

Why Zoo? “Because Coordinating Distributed Systems is a Zoo”.

In this post I’m going show how to install and integrate Apache ZooKeeper with PHP. We will use the service to coordinate independent PHP scripts and let them agree on which one is going to be the leader (so called leader election). When the leader exit (or crash) workers should detect it and elect a new one.

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications. Each time they are implemented there is a lot of work that goes into fixing the bugs and race conditions that are inevitable. Because of the difficulty of implementing these kinds of services, applications initially usually skimp on them ,which make them brittle in the presence of change and difficult to manage. Even when done correctly, different implementations of these services lead to management complexity when the applications are deployed.

ZooKeeper is a Java application but it also comes with C bindings. There is a PHP extension created and maintained by Andrei Zmievski since 2009. You can download it from PECL or get directly from GitHub PHP-ZooKeeper.

To get started with the extension you need to install ZooKeeper. Download it from the official site.

$ tar zxfv zookeeper-3.4.5.tar.gz
$ cd zookeeper-3.4.5/src/c
$ ./configure --prefix=/usr/
$ make
$ sudo make install

That will install ZooKeeper’s library and headers. Now you are ready to compile the PHP extension.

$ cd
$ git clone https://github.com/andreiz/php-zookeeper.git
$ cd php-zookeeper
$ phpize
$ ./configure
$ make
$ sudo make install

Add “zookeeper.so” to PHP configuration.

$ vim /etc/php5/cli/conf.d/20-zookeeper.ini

I edit only CLI config because I won’t need it in a web server context. Paste the below line into the ini file.

extension=zookeeper.so

Make sure the extension is working.

$ php -m | grep zookeeper
zookeeper

It’s a good time to run ZooKeeper. The only missing thing is configuration. Create a directory for the service where it can keep all its data.

$ mkdir /home/you-account/zoo
$ cd
$ cd zookeeper-3.4.5/
$ cp conf/zoo_sample.cfg conf/zoo.cfg
$ vim conf/zoo.cfg

Find attribute called “dataDir” and point it to your “/home/you-account/zoo” directory.

$ bin/zkServer.sh start
$ bin/zkCli.sh -server 127.0.0.1:2181
[zk: 127.0.0.1:2181(CONNECTED) 14] create /test 1
Created /test
[zk: 127.0.0.1:2181(CONNECTED) 19] ls /
[test, zookeeper]

That will connect you to the service and create a “/test” znode (we will use it in a second). ZooKeeper stores data in a tree structure. It’s very similar to a file system with a difference that “directories” can simultaneously behave like files. Every entity stored by ZooKeeper is called znode. Node in an ambiguous word in this context so to avoid confusion the system is using different name.

Leave the client connected because we will use it in a second. Open a new window and create a zookeeperdemo1.php file.

get( '/test', array($this, 'watcher' ) );
  }

}

$zoo = new ZookeeperDemo('127.0.0.1:2181');
$zoo->get( '/test', array($zoo, 'watcher' ) );

while( true ) {
  echo '.';
  sleep(2);
}

Now run the script.

$ php zookeeperdemo1.php

It should produce a dot every 2 seconds. Now switch to ZooKeeper client and update “/test” value.

[zk: 127.0.0.1:2181(CONNECTED) 20] set /test foo

That should immaterially trigger “Insider Watcher” message in the PHP script. How did that happen?

ZooKeeper provides watchers which can be attached to znodes. If watched znode change the service will instantly inform all interested clients about it. This is how the PHP script knew about the change. Second parameter of Zookeeper::get method is callback. Watcher gets consumed when event is triggered so we need to set it again in the callback.

Now you are ready to create a distributed application. The challenge is to let independent programs decide which one should be coordinating them (the leader) and which should be doing the job (workers). The process is called leader election and you can read about implementation at ZooKeeper Recipes and Solutions.
In a nutshell each process looks at a process next to it. If a watched process exit or crash the watching program should check is it the oldest process. If it is it will become the leader.

In real life application the leader should be allocating tasks to workers, monitor progress and store results. I will skip this part for the sake of simplicity.

Create a new PHP file and call it worker.php.

 Zookeeper::PERM_ALL,
                      'scheme' => 'world',
                      'id' => 'anyone' ) );

  private $isLeader = false;

  private $znode;

  public function __construct( $host = '', $watcher_cb = null, $recv_timeout = 10000 ) {
    parent::__construct( $host, $watcher_cb, $recv_timeout );
  }

  public function register() {
    if( ! $this->exists( self::CONTAINER ) ) {
      $this->create( self::CONTAINER, null, $this->acl );
    }

    $this->znode = $this->create( self::CONTAINER . '/w-', 
                                  null, 
                                  $this->acl, 
                                  Zookeeper::EPHEMERAL | Zookeeper::SEQUENCE );

    $this->znode = str_replace( self::CONTAINER .'/', '', $this->znode );

    printf( "I'm registred as: %sn", $this->znode );

    $watching = $this->watchPrevious();

    if( $watching == $this->znode ) {
      printf( "Nobody here, I'm the leadern" );
      $this->setLeader( true );
    }
    else {
      printf( "I'm watching %sn", $watching );
    }
  }

  public function watchPrevious() {
    $workers = $this->getChildren( self::CONTAINER );
    sort( $workers );
    $size = sizeof( $workers );
    for( $i = 0 ; $i znode == $workers[ $i ] ) {
        if( $i > 0 ) {
          $this->get( self::CONTAINER . '/' . $workers[ $i - 1 ], array( $this, 'watchNode' ) );
          return $workers[ $i - 1 ];
        }

        return $workers[ $i ];
      }
    }

    throw new Exception(  sprintf( "Something went very wrong! I can't find myself: %s/%s", 
                          self::CONTAINER, 
                          $this->znode ) );
  }

  public function watchNode( $i, $type, $name ) {
    $watching = $this->watchPrevious();
    if( $watching == $this->znode ) {
      printf( "I'm the new leader!n" );
      $this->setLeader( true );
    }
    else {
      printf( "Now I'm watching %sn", $watching );
    }
  }

  public function isLeader() {
    return $this->isLeader;
  }

  public function setLeader($flag) {
    $this->isLeader = $flag;
  }

  public function run() {
    $this->register();

    while( true ) {
      if( $this->isLeader() ) {
        $this->doLeaderJob();
    }
    else {
      $this->doWorkerJob();
    }

      sleep( 2 );
    }
  }

  public function doLeaderJob() {
    echo "Leadingn";
  }

  public function doWorkerJob() {
    echo "Workingn";
  }

}

$worker = new Worker( '127.0.0.1:2181' );
$worker->run();

Open at least 3 terminals and run the script in each of them.

# term1

$ php worker.php 
I'm registred as: w-0000000001
Nobody here, I'm the leader
Leading

# term2

$ php worker.php 
I'm registred as: w-0000000002
I'm watching w-0000000001
Working

# term3

$ php worker.php 
I'm registred as: w-0000000003
I'm watching w-0000000002
Working

Now simulate crash of the leader. Exit first script with Ctrl+c or any other method. Nothing will change for few seconds. Workers will happily continue working. Eventually ZooKeeper will discover timeout and new leader is going to be elected.

It’s easy to understand the script but it might be worth to comment on used Zookeeper flags.

$this->znode = $this->create( self::CONTAINER . '/w-', 
                              null, 
                              $this->acl, 
                              Zookeeper::EPHEMERAL | Zookeeper::SEQUENCE );

Every znode is as EPHEMERAL and SEQUENCE.

EPHEMERAL means that znode will be removed when client disconnect. This is how the PHP script knew about timeout. SEQUENCE means that a sequence string is going to be append to every znode name. We used them as unique identifiers for workers.

Be aware there are some problems on PHP side. The extension is in beta version and If you not follow certain patterns it’s quite easy to get segmentation faults. For example, I wasn’t able to pass an ordinary function as a callback. It has to be a method. The good news is that if something is working it should remain in that state. I hope more people from PHP community will get excited about Apache ZooKeeper and the extension will receive more support.

ZooKeeper is great software with clean and simple API. Thanks to quality documentation, examples and recipes anybody can start writing distributed software. Give it a go, it’s fun!

Useful links:

– Yahoo research on ZooKeeper. Very good read with examples of real life applications. If you had to read only one thing about the service that would be it.
– PHP ZooKeeper
– PHP ZooKeeper API
– PHP ZooKeeper example