New versions of NeuroBrix, our open-source inference engine, and of nbxtensor, its tensor library, are on PyPI. Everything below came from running real prompts on our own engine.

Long prompts are tokenized in linear time, with the same tokens

A 12,265-token prompt to a Llama-tokenizer model took 25.7 s to tokenize before the model saw a single token. It now takes 0.03 s, and produces exactly the same tokens.

A model that ships thousands of added tokens stalled on long prompts too: on a 34,880-character prompt it went from 43.6 s to 0.02 s. And neurobrix serve no longer keeps every long prompt it has seen in memory.

A new server does not tune its kernels twice

NeuroBrix measures the best kernel settings for each machine and stores them. Until now, only the kernels loaded before the first computation received those stored settings; the ones a server loads at its first request were measured again by every new process. Now every kernel gets this machine's settings from the start.

Errors with names

  • neurobrix run refuses a prompt longer than the model's context window by name, before anything loads, as serve already did. It used to fail inside a GPU kernel or abort the process.
  • A run whose GPU reports its error late exits with that error named (rc 1) instead of aborting.
  • /v1/audio/speech without a voice, for a model that ships several voices and declares no default, now returns a 400 naming voice and the available voices, no longer a 500.
  • neurobrix run --turns reports each turn's own timings.

The engine calls its library

The engine now leaves to nbxtensor what belongs to a tensor library: the replay of decode steps, and the expert store of mixture-of-experts models (its device slots, copies, and routing over several GPUs). We checked this the only way that counts: the outputs are byte-identical to the previous version on our gate's models, on one GPU and on two, with both engines.

The terminal coding agent left the engine: coding agents become their own project and talk to neurobrix serve through its OpenAI-compatible API.

Try it 

pip install -U neurobrix # Triton engine, no PyTorch pip install -U "neurobrix[pytorch]" # with the PyTorch engine

Full release notes: v0.6.10 on GitHub. If NeuroBrix is useful to you, a ⭐ on GitHub is the signal we read.