A 125-million-parameter Llama-style language model, pretrained from random weights on a legal and financial corpus. Give it the start of a sentence and it continues in the legal register.
This is a base model, not a chatbot. It was trained on next-token prediction only, so it continues text rather than answering questions. Prompt it with the opening of a sentence and watch it complete the thought.
The honest quality metric is held-out validation perplexity: 10.87 (lower is better), measured over a 1% held-out split (~22M tokens). It speaks the legal register (case-citation phrasing, procedural language) but it does not know facts — at 125M parameters a model holds only about 31MB of usable knowledge. Grounded facts would need retrieval (RAG).
| architecture | Llama-style decoder (RoPE, SwiGLU, RMSNorm, MHA) |
| layers / hidden / heads | 12 / 768 / 12 (head dim 64), kv-heads 12 |
| parameters | 125,847,552 (tied embeddings) |
| vocab / context | 16,384 byte-level BPE / 1,024 tokens |
| train / val tokens | 2.18B / 22.0M (99/1 split) |
| corpus mix | US case law ~33% · SEC filings ~39% · edu web ~28% |
| optimizer | AdamW (0.9, 0.95), wd 0.1, grad-clip 1.0 |
| lr schedule | 6e-4 → 6e-5 cosine, 200M-token warmup |
| batch / precision | 524,288 tokens/step, bf16 |
| compute | 8×H100 DDP · 1 epoch · ~24 min |
| final val perplexity | 10.87 |
Corpus: US case law, SEC filings, educational web text. First call may take ~15–30s while the CPU endpoint wakes from idle.