libs/PHPCrawler.class.php
public function goMultiProcessed($process_count = 3, $multiprocess_mode = 1)
  { 
    $this->multiprocess_mode = $multiprocess_mode;
    
    // Check if fork is supported
    if (!function_exists("pcntl_fork"))
    {
      throw new Exception("PHPCrawl running with multi processes not supported in this PHP-environment (function pcntl_fork() missing).".
                          "Try running from command-line (cli) and/or installing the PHP PCNTL-extension.");
    }
    
    if (!function_exists("sem_get"))
    {
      throw new Exception("PHPCrawl running with multi processes not supported in this PHP-environment (function sem_get() missing).".
                          "Try installing the PHP SEMAPHORE-extension.");
    }
    
    if (!function_exists("posix_kill"))
    {
      throw new Exception("PHPCrawl running with multi processes not supported in this PHP-environment (function posix_kill() missing).".
                          "Try installing the PHP POSIX-extension.");
    }
    
    if (!class_exists("PDO"))
    {
      throw new Exception("PHPCrawl running with multi processes not supported in this PHP-environment (class PDO missing).".
                          "Try installing the PHP PDO-extension.");
    }
    
    PHPCrawlerBenchmark::start("crawling_process");
    
    // Set url-cache-type to sqlite.
    $this->url_cache_type = PHPCrawlerUrlCacheTypes::URLCACHE_SQLITE;
    
    // Init process
    $this->initCrawlerProcess();
    
    // Process robots.txt
    if ($this->obey_robots_txt == true)
      $this->processRobotsTxt();
    
    // Fork off child-processes
    $pids = array();
    
    for($i=1; $i<=$process_count; $i++)
    {
      $pids[$i] = pcntl_fork();

      if(!$pids[$i])
      {   
        // Childprocess goes here
        $this->is_chlid_process = true;
        $this->child_process_number = $i;
        $this->ProcessCommunication->registerChildPID(getmypid());
        $this->startChildProcessLoop();
      }
    }
        
    // Set flag "parent-process"
    $this->is_parent_process = true;
    
    // Determinate all child-PIDs
    $this->child_pids = $this->ProcessCommunication->getChildPIDs($process_count);
    
    // If crawler runs in MPMODE_PARENT_EXECUTES_USERCODE-mode -> start controller-loop
    if ($this->multiprocess_mode == PHPCrawlerMultiProcessModes::MPMODE_PARENT_EXECUTES_USERCODE)
    {
      $this->starControllerProcessLoop();
    }
     
    // Wait for childs to finish
    for ($i=1; $i<=$process_count; $i++)
    {
      pcntl_waitpid($pids[$i], $status, WUNTRACED);
    }
    
    // Get crawler-status (needed for process-report)
    $this->crawlerStatus = $this->ProcessCommunication->getCrawlerStatus();
    
    // Cleanup crawler
    $this->cleanup();
    
    PHPCrawlerBenchmark::stop("crawling_process");
  }
public function goMultiProcessed($process_count = 3, $multiprocess_mode = 1)
  { 
    $this->multiprocess_mode = $multiprocess_mode;
    
    // Check if fork is supported
    if (!function_exists("pcntl_fork"))
    {
      throw new Exception("PHPCrawl running with multi processes not supported in this PHP-environment (function pcntl_fork() missing).".
                          "Try running from command-line (cli) and/or installing the PHP PCNTL-extension.");
    }
    
    if (!function_exists("sem_get"))
    {
      throw new Exception("PHPCrawl running with multi processes not supported in this PHP-environment (function sem_get() missing).".
                          "Try installing the PHP SEMAPHORE-extension.");
    }
    
    if (!function_exists("posix_kill"))
    {
      throw new Exception("PHPCrawl running with multi processes not supported in this PHP-environment (function posix_kill() missing).".
                          "Try installing the PHP POSIX-extension.");
    }
    
    if (!class_exists("PDO"))
    {
      throw new Exception("PHPCrawl running with multi processes not supported in this PHP-environment (class PDO missing).".
                          "Try installing the PHP PDO-extension.");
    }
    
    PHPCrawlerBenchmark::start("crawling_process");
    
    // Set url-cache-type to sqlite.
    $this->url_cache_type = PHPCrawlerUrlCacheTypes::URLCACHE_SQLITE;
    
    // Init process
    $this->initCrawlerProcess();
    
    // Process robots.txt
    if ($this->obey_robots_txt == true)
      $this->processRobotsTxt();
    
    // Fork off child-processes
    $pids = array();
    
    for($i=1; $i<=$process_count; $i++)
    {
      $pids[$i] = pcntl_fork();

      if(!$pids[$i])
      {   
        // Childprocess goes here
        $this->is_chlid_process = true;
        $this->child_process_number = $i;
        $this->ProcessCommunication->registerChildPID(getmypid());
        $this->startChildProcessLoop();
      }
    }
        
    // Set flag "parent-process"
    $this->is_parent_process = true;
    
    // Determinate all child-PIDs
    $this->child_pids = $this->ProcessCommunication->getChildPIDs($process_count);
    
    // If crawler runs in MPMODE_PARENT_EXECUTES_USERCODE-mode -> start controller-loop
    if ($this->multiprocess_mode == PHPCrawlerMultiProcessModes::MPMODE_PARENT_EXECUTES_USERCODE)
    {
      $this->starControllerProcessLoop();
    }
     
    // Wait for childs to finish
    for ($i=1; $i<=$process_count; $i++)
    {
      pcntl_waitpid($pids[$i], $status, WUNTRACED);
    }
    
    // Get crawler-status (needed for process-report)
    $this->crawlerStatus = $this->ProcessCommunication->getCrawlerStatus();
    
    // Cleanup crawler
    $this->cleanup();
    
    PHPCrawlerBenchmark::stop("crawling_process");
  }
Starts the cralwer by using multi processes.

When using this method instead of the {@link go()}-method to start the crawler, phpcrawl will use the given
number of processes simultaneously for spidering the target-url.
Using multi processes will speed up the crawling-progress dramatically in most cases.

There are some requirements though to successfully run the cralwler in multi-process mode:
<ul>
<li> The multi-process mode only works on unix-based systems (linux)</li>
<li> Scripts using the crawler have to be run from the commandline (cli)</li>
<li> The <a href="http://php.net/manual/en/pcntl.installation.php">PCNTL-extension</a> for php (process control) has to be installed and activated.</li>
<li> The <a href="http://php.net/manual/en/sem.installation.php">SEMAPHORE-extension</a> for php has to be installed and activated.</li>
<li>The <a href="http://de.php.net/manual/en/posix.installation.php">POSIX-extension</a> for php has to be installed and activated.</li>
<li> The <a href="http://de2.php.net/manual/en/pdo.installation.php">PDO-extension</a> together with the SQLite-driver (PDO_SQLITE) has to be installed and activated.</li>
</ul>

PHPCrawls supports two different modes of multiprocessing:
<ol>
<li><b>{@link PHPCrawlerMultiProcessModes}::MPMODE_PARENT_EXECUTES_USERCODE</b>

The cralwer uses multi processes simultaneously for spidering the target URL, but the usercode provided to
the overridable function {@link handleDocumentInfo()} gets always executed on the same main-process. This
means that the <b>usercode never gets executed simultaneously</b> and so you dont't have to care about
concurrent file/database/handle-accesses or smimilar things.
But on the other side the usercode may slow down the crawling-procedure because every child-process has to
wait until the usercode got executed on the main-process. <b>This ist the recommended multiprocess-mode!</b>
</li>
<li><b>{@link PHPCrawlerMultiProcessModes}::MPMODE_CHILDS_EXECUTES_USERCODE</b>

The cralwer uses multi processes simultaneously for spidering the target URL, and every chld-process executes
the usercode provided to the overridable function {@link handleDocumentInfo()} directly from it's process. This
means that the <b>usercode gets executed simultaneously</b> by the different child-processes and you should
take care of concurrent file/data/handle-accesses proberbly (if used).

When using this mode and you use any handles like database-connections or filestreams in your extended
crawler-class, you should open them within the overridden mehtod {@link initChildProcess()} instead of opening
them from the constructor. For more details see the documentation of the {@link initChildProcess()}-method.
</li>
</ol>

Example for starting the crawler with 5 processes using the recommended MPMODE_PARENT_EXECUTES_USERCODE-mode:
<code>
$crawler->goMultiProcessed(5, PHPCrawlerMultiProcessModes::MPMODE_PARENT_EXECUTES_USERCODE);
</code>

Please note that increasing the number of processes to high values does't automatically mean that the crawling-process
will go off faster! Using 3 to 5 processes should be good values to start from.

@param int $process_count     Number of processes to use
@param int $multiprocess_mode The multiprocess-mode to use.
One of the {@link PHPCrawlerMultiProcessModes}-constants
@section 1 Basic settings