danny50610/bpe-tokeniser

PHP port for openai/tiktoken (most)

Maintainers

Details

github.com/danny50610/bpe-tokeniser

Source

Issues

Installs: 123 235

Dependents: 1

Suggesters: 0

Security: 0

Stars: 6

Watchers: 2

Forks: 0

Open Issues: 0

0.2.0 2024-05-18 14:04 UTC

Requires

  • php: ^8.1

Requires (Dev)

MIT abdaec2e7ee729073fa345c987b81397cd790932

  • Chen, Danny <danny50610.woop@yahoo.com.tw>

tokenizergpt-4gpt-3.5tiktokencl100k_base

This package is auto-updated.

Last update: 2024-11-09 07:24:19 UTC

README

PHP Test codecov Latest Stable Version Total Downloads License

PHP port for openai/tiktoken (most)

Supported encodings

  • gpt-3.5-turbo
  • gpt-4
  • gpt-4o
  • more ...

For available encodings, see src/EncodingFactory.php

Installation

composer require danny50610/bpe-tokeniser

Example

GPT-4 / GPT-3.5-Turbo (cl100k_base)

use Danny50610\BpeTokeniser\EncodingFactory;

$enc = EncodingFactory::createByEncodingName('cl100k_base');

var_dump($enc->encode("hello world"));
/**
 * output: 
 * array(2) {
 *  [0]=>
 *  int(15339)
 *  [1]=>
 *  int(1917)
 * }
 */

var_dump($enc->decode($enc->encode("hello world")));
// output: string(11) "hello world"
use Danny50610\BpeTokeniser\EncodingFactory;

$enc = EncodingFactory::createByModelName('gpt-3.5-turbo');

var_dump($enc->decode($enc->encode("hello world")));
// output: string(11) "hello world"