BIAS combines cutting-edge multi-format support with production-grade reliability and performance.
BIAS has evolved from a JSON optimizer into a universal data gateway for LLMs with 8 complete format adapters, comprehensive test coverage (100% success rate), and extensive validation with real-world data. Automatic format detection and pluggable adapter architecture make it production-ready for any use case.
Full JSON support with schema inference, nested objects/arrays, type preservation, and deterministic encoding. The most mature adapter with comprehensive test coverage.
CommonMark + GitHub Flavored Markdown (GFM) support with headings, lists, code blocks, tables, links, images, emphasis, and strikethrough. Full round-trip preservation.
Advanced HTML adapter with DOCTYPE preservation, semantic elements, attributes, text/comment nodes, and reconstruction hints. Handles real-world HTML documents.
Complete YAML 1.2 support with document markers (---), block/flow styles, anchors & aliases, null value distinction, and reserved namespace validation.
XML 1.0 support with elements, attributes, CDATA, processing instructions, and namespaces. Designed for structured data and configuration files.
Full TOML 0.5 support with section headers [tables], nested tables via dot notation, arrays of tables [[posts]], and complete type preservation (datetime, int, float, boolean).
CSV/TSV support with header detection, type inference, quoted fields, and multi-line values. Perfect for tabular data and analytics exports.
JSON-RPC 2.0 protocol support for request/response/notification messages. Specialized adapter for RPC communication patterns.
BIAS automatically detects input formats using confidence-based scoring (0.0-1.0 scale). No manual format specification required—just pass your data.
1. Sample input data (first 1024 bytes for quick detection)
2. Run all registered adapter detectors in parallel
3. Each adapter returns confidence score (0.0-1.0):
- 0.99: Document markers (YAML ---, JSON {})
- 0.95: Section headers (TOML [section])
- 0.90: Key-value patterns
- 0.80: Structural hints
- 0.50: Weak indicators
4. Select adapter with highest confidence above threshold
5. Fall back to comprehensive detection if needed
| Format | Primary Indicators | Confidence |
|---|---|---|
| JSON | Starts with { or [ |
0.99 |
| YAML | Document markers ---, directives %YAML |
0.99 |
| TOML | Section headers [section] |
0.95 |
| HTML | <!DOCTYPE html>, <html> tags |
0.99 |
| Markdown | Headings #, lists -, code blocks ``` |
0.50-0.80 |
50-150µs average encoding time for typical payloads. Streaming design with minimal allocations. Memory-efficient processing handles documents up to 100MB+.
30-80µs average decoding time. 4.3x faster than v0.1. Optimized graph traversal with zero-copy string handling where possible.
Built-in guards: Max depth 128, Max entities 100K. Prevents resource exhaustion attacks. Configurable limits for your use case.
Thread-safe design with no global state. Process multiple documents in parallel. Perfect for high-throughput server deployments.
| Operation | Small (1KB) | Medium (10KB) | Large (100KB) |
|---|---|---|---|
| JSON Detection | ~5µs | ~8µs | ~12µs |
| JSON → Graph | ~45µs | ~180µs | ~2.1ms |
| Graph → BIAS | ~30µs | ~120µs | ~1.4ms |
| Full Roundtrip | <100µs | ~350µs | ~4.2ms |
| Token Savings | 60-63% | 48-52% | 25-28% |
Extensive test suite covering unit tests, integration tests, corpus validation, and benchmarks. Every adapter thoroughly tested for correctness and performance.
Perfect round-trip conversion guaranteed. Every structure, every value, every semantic meaning preserved. Deterministic encoding for reproducibility.
_bias_* prefix reserved for internal types/predicates.
Automatic validation prevents conflicts. Future-proof for new features.
Comprehensive error types: parse errors, conversion errors, validation errors. Clear error messages with context. Graceful degradation where appropriate.
Validated across Gemini, Claude, GPT-4, Llama, Groq, Cerebras. Models successfully consume BIAS with short instruction prompts.
Written in Rust with strong type guarantees. NewType pattern for IDs. Memory safety without garbage collection overhead.
Production-ready gRPC server (biasd) with health checks, metrics,
and streaming support. Easy to deploy and scale.
Official bindings for Python, TypeScript, JavaScript. Simple API, drop-in replacement for JSON.stringify/parse.
Command-line tools for encoding, decoding, validation, and benchmarking. Perfect for scripting and automation.
React/TypeScript web UI for interactive demos. Real-time encoding/decoding with visual token comparison.
BIAS uses a unified Graph v2 representation as the canonical intermediate format. All adapters convert to/from this graph for maximum flexibility.
pub struct Graph {
entities: Vec<Entity>, // Nodes (objects, arrays, values)
predicates: Vec<Predicate>, // Edges (relationships)
literals: Vec<String>, // String pool (deduplicated)
}
pub struct Entity {
kind: NodeKind, // Object, Array, Literal, Synthetic
id: EntityId, // Unique identifier
}
pub struct Predicate {
subject: EntityId, // Source node
verb: LiteralId, // Relationship name
object: EntityId, // Target node
order: u32, // Sequence order
}
See real-world examples and performance benchmarks.