# Tokoscope

> Tokoscope is the token optimization platform for teams building on LLMs. It audits, compresses, and monitors LLM token usage across OpenAI, Anthropic, and Gemini through a lightweight SDK and real-time dashboard.

## Product Overview

Tokoscope wraps existing LLM clients in two lines of code with zero infrastructure changes. It sits between your application and the LLM API, automatically detecting token waste, compressing bloated prompts, caching semantically similar requests, and providing real-time cost attribution by endpoint, user, and provider.

The core insight: 40–70% of tokens in the average production prompt are pure waste — redundant instructions, repeated phrases, stuffed context windows. Tokoscope surfaces this waste and eliminates it automatically.

**Key differentiators:**
- Two-line SDK integration — wrap your existing client, all calls work unchanged
- Automatic prompt compression using Claude Haiku (typical 40–90% reduction)
- Two-layer semantic caching: exact match + OpenAI embedding similarity (85%+ threshold)
- Per-user token attribution for multi-tenant applications
- Supports OpenAI, Anthropic, and Gemini natively
- Available as both npm (JavaScript) and PyPI (Python) packages

---

## SDK Integration

### JavaScript — OpenAI
```javascript
import OpenAI from 'openai'
import { wrap } from 'tokoscope'

const client = wrap(new OpenAI(), {
  apiKey: 'ts_live_...',  // from app.tokoscope.com/settings
  userId: 'user_123'      // optional: per-user attribution
})

// All existing calls work unchanged
const response = await client.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'Hello' }]
})
```

### JavaScript — Anthropic
```javascript
import Anthropic from '@anthropic-ai/sdk'
import { wrap } from 'tokoscope'

const client = wrap(new Anthropic(), { apiKey: 'ts_live_...' })

const response = await client.messages.create({
  model: 'claude-sonnet-4-6',
  max_tokens: 1024,
  messages: [{ role: 'user', content: 'Hello' }]
})
```

### JavaScript — Gemini
```javascript
import { GoogleGenerativeAI } from '@google/generative-ai'
import { wrap } from 'tokoscope'

const genAI = wrap(new GoogleGenerativeAI(GEMINI_KEY), { apiKey: 'ts_live_...' })
const model = genAI.getGenerativeModel({ model: 'gemini-2.5-flash' })
const result = await model.generateContent('Hello')
```

### Python — OpenAI
```python
from openai import OpenAI
from tokoscope import wrap

client = wrap(OpenAI(), api_key='ts_live_...', user_id='user_123')

response = client.chat.completions.create(
    model='gpt-4o',
    messages=[{'role': 'user', 'content': 'Hello'}]
)

# Async support
response = await client.chat.completions.acreate(
    model='gpt-4o',
    messages=[{'role': 'user', 'content': 'Hello'}]
)
```

### Python — Anthropic
```python
from anthropic import Anthropic
from tokoscope import wrap

client = wrap(Anthropic(), api_key='ts_live_...')

response = client.messages.create(
    model='claude-sonnet-4-6',
    max_tokens=1024,
    messages=[{'role': 'user', 'content': 'Hello'}]
)
```

### Python — Gemini
```python
import google.generativeai as genai
from tokoscope import wrap

model = wrap(genai.GenerativeModel('gemini-2.5-flash'), api_key='ts_live_...')
result = model.generate_content('Hello')
```

---

## wrap() API Reference

### JavaScript
```javascript
wrap(client, options)
```

| Option | Type | Required | Description |
|---|---|---|---|
| apiKey | string | Yes | Tokoscope API key from app.tokoscope.com/settings |
| userId | string | No | End-user ID for per-user attribution |

### Python
```python
wrap(client, api_key, user_id=None)
```

| Parameter | Type | Required | Description |
|---|---|---|---|
| api_key | string | Yes | Tokoscope API key |
| user_id | string | No | End-user ID for per-user attribution |

---

## Features In Detail

### Prompt Compression
Every prompt is scored for token waste using a proprietary scoring algorithm that detects:
- Repeated or redundant instructions
- Verbose phrasing that can be condensed
- Unnecessary context padding
- Common filler phrases ("please note that", "it is important to", "as an AI")

Prompts scoring above 30% waste are automatically rewritten by Claude Haiku to their minimum effective form. The original and compressed versions are both stored for comparison in the dashboard.

**Example:**
- Original: 113 tokens — "Please note that it is very important that you make sure to respond to my question..."
- Compressed: 8 tokens — "What is the capital of France? Answer concisely."
- Savings: 90% token reduction, identical model output

### Semantic Caching
Two-layer caching system:

**Layer 1 — Exact match:**
Prompts are hashed (model + content). Identical prompts return cached responses instantly with zero API calls.

**Layer 2 — Semantic match:**
New prompts are embedded using OpenAI's `text-embedding-3-small` model. Embeddings are compared against cached embeddings using cosine similarity. If similarity ≥ 85%, the cached response is returned.

Cache TTL: 7 days. Cache can be cleared from the dashboard Settings page.

Console output on cache hit:
```
⚡ Tokoscope cache hit [semantic (89.3% match)] — saved 93 tokens ($0.000049)
⚡ Tokoscope cache hit [exact] — saved 21 tokens ($0.000006)
```

### Cost Attribution
Every API call is logged with:
- Provider (openai, anthropic, gemini)
- Model name
- Input token count
- Output token count
- Total cost (calculated using current provider pricing)
- Waste score (0–100)
- Endpoint identifier
- End user ID (if provided)
- Cache hit status and savings
- Compressed prompt (if compression was applied)

### Per-User Tracking
Pass a `userId` (JS) or `user_id` (Python) to attribute token usage to individual end users. The Users dashboard shows:
- Full paginated list of users sorted by token usage
- Per-user waste score
- Per-user cost breakdown
- Recent API calls per user
- Search by user ID

### Budget Alerts
Set a monthly spend threshold in Settings. When cumulative spend crosses the threshold, an email alert is sent via Loops. Alerts fire once per calendar month per user. The email includes current monthly spend and the threshold that was crossed.

---

## Supported Models & Pricing

### OpenAI
| Model | Input (per 1K tokens) | Output (per 1K tokens) |
|---|---|---|
| gpt-4o | $0.005 | $0.015 |
| gpt-4o-mini | $0.00015 | $0.0006 |
| gpt-4-turbo | $0.01 | $0.03 |
| gpt-3.5-turbo | $0.0005 | $0.0015 |

### Anthropic
| Model | Input (per 1K tokens) | Output (per 1K tokens) |
|---|---|---|
| claude-opus-4-6 | $0.015 | $0.075 |
| claude-sonnet-4-6 | $0.003 | $0.015 |
| claude-haiku-4-5 | $0.00025 | $0.00125 |

### Gemini
| Model | Input (per 1K tokens) | Output (per 1K tokens) |
|---|---|---|
| gemini-2.5-flash | $0.0001 | $0.0004 |
| gemini-2.5-pro | $0.00125 | $0.01 |
| gemini-1.5-flash | $0.000075 | $0.0003 |
| gemini-1.5-pro | $0.00125 | $0.005 |
| gemini-2.0-flash | $0.0001 | $0.0004 |

---

## Pricing Plans

| Plan | Price | Tokens monitored | Features |
|---|---|---|---|
| Free | $0/month | 500K tokens/month | Dashboard, waste scoring, exact caching, OpenAI + Anthropic + Gemini |
| Pro | $49/month per workspace | Unlimited | + Prompt compression, semantic caching, per-user tracking, budget alerts, 5 workspaces |
| Team | $99/month + usage | Unlimited | + Unlimited workspaces, team access, Slack/webhook alerts, CSV export, priority support |

---

## Dashboard Pages

- `/dashboard` — Token usage charts, cost stats, cache hit rate, top endpoints
- `/prompts` — All tracked prompts with waste scores, compressed versions, suggestions
- `/users` — Per-user token usage, cost, waste scores, recent calls
- `/onboarding` — SDK setup guide with copy-paste code for all providers
- `/settings` — API key management, budget alert threshold, cache management

---

## Technical Architecture

- **SDK:** Wraps LLM client methods, intercepts API calls, sends tracking events to Firebase Cloud Functions
- **Cloud Functions:** Firebase Functions (Node.js 22) — trackEvent, checkCache, clearCache, getCacheStats, joinWaitlist
- **Database:** Firestore — events, cache, users, waitlist collections
- **Email:** Loops.so — welcome, drip, budget alert, referral reward sequences
- **Embeddings:** OpenAI text-embedding-3-small for semantic cache comparison
- **Compression:** Anthropic Claude Haiku for prompt rewriting
- **Hosting:** Firebase Hosting — tokoscope.com (landing) and app.tokoscope.com (dashboard)

---

## SDK Versions

| Version | JS | Python | Key features |
|---|---|---|---|
| 0.7.0 | ✓ | — | Anthropic + Gemini streaming (JS) |
| 0.8.0 | — | ✓ | Anthropic + Gemini streaming (Python): create(stream=True), .stream() context manager, generate_content(stream=True) |
| 0.6.0 | — | ✓ | Semantic caching, async support (acreate) |
| 0.5.0 | ✓ | — | Semantic caching (85% threshold) |
| 0.4.0 | ✓ | ✓ | Gemini support |
| 0.3.0 | ✓ | ✓ | 7-day exact match caching |
| 0.2.0 | ✓ | ✓ | Per-user tracking (userId/user_id) |
| 0.1.0 | ✓ | ✓ | Initial release: OpenAI + Anthropic, tracking, compression, budget alerts |

---

## Articles

Tokoscope publishes technical articles on LLM cost optimization, local models, and the broader LLM tooling ecosystem.

### Cost Optimization
- We analyzed thousands of LLM API calls (https://tokoscope.com/articles/llm-token-waste): Where token waste hides in production apps. The three biggest sources are bloated system prompts (~70% waste), repeated requests without caching (~60%), and stuffed context windows (~50%).
- Semantic caching: why exact match isn't enough (https://tokoscope.com/articles/semantic-caching): How the two-layer cache works — exact hash match first, then semantic similarity at an 85% cosine threshold using OpenAI embeddings.
- How to reduce OpenAI API costs by 60% (https://tokoscope.com/articles/reduce-openai-costs): Practical walkthrough of prompt compression, semantic caching, and cost attribution as compounding techniques.
- Getting started in 5 minutes (https://tokoscope.com/articles/getting-started): Step-by-step integration for OpenAI, Anthropic, and Gemini.

### Models & Providers
- GLM 5.2 and the 1M token context window (https://tokoscope.com/articles/glm-5-2): The most powerful open-weight LLM of 2026 and how a 1M token context window affects API cost.
- DeepSeek: frontier performance at a fraction of the cost (https://tokoscope.com/articles/deepseek): DeepSeek's MoE, MLA, GRPO, and FP8 innovations. API pricing is 10-18x cheaper than frontier alternatives on input tokens.
- LLM benchmarks explained (https://tokoscope.com/articles/llm-benchmarks): What MMLU, HumanEval, GPQA, SWE-bench, and LMSYS Arena measure, with cost-per-benchmark-point analysis. Gemini 2.5 Flash leads on cost efficiency.
- RLHF explained (https://tokoscope.com/articles/rlhf): How RLHF makes models helpful but verbose (reward hacking toward longer outputs), and why that verbosity bias increases token costs. Covers DPO, GRPO, and Constitutional AI alternatives.
- OpenAI Codex: what replaced it (https://tokoscope.com/articles/codex): Codex history and the modern coding model landscape, including agentic coding token costs.

### Local & Open-Source LLMs
- How to run a local LLM (https://tokoscope.com/articles/run-local-llm): Running Ollama, LM Studio, and llama.cpp, with an honest local-vs-cloud cost comparison. Local models are free per-token but have hardware, electricity, and maintenance costs.
- LLM Studio vs Ollama vs LM Studio (https://tokoscope.com/articles/llm-studio): Disambiguates LM Studio (desktop app), H2O LLM Studio (fine-tuning platform), Ollama, and llama.cpp.
- Best local LLM for coding in 2026 (https://tokoscope.com/articles/best-local-llm-coding): Ranked coding models — Qwen2.5-Coder-32B (best overall), DeepSeek-Coder-V2, CodeLlama-70B, Mistral-7B — with hardware requirements.
- Best Ollama models in 2026 (https://tokoscope.com/articles/ollama-models): Best Ollama models by use case (chat, coding, reasoning, embeddings) with a size-to-hardware guide.
- Hugging Face: the GitHub of AI models (https://tokoscope.com/articles/hugging-face): The Hub, Transformers library, Inference API, Endpoints, and Spaces, plus a cost comparison with cloud APIs.

### LLM Infrastructure & Tooling
- What is an LLM gateway? (https://tokoscope.com/articles/llm-gateway): What gateways do (routing, caching, cost control) and when SDK wrapping is the simpler alternative.
- OpenRouter: one API for 100+ LLMs (https://tokoscope.com/articles/openrouter): Tradeoffs of unified routing (latency, markup, dependency) and how to layer Tokoscope on top.
- LiteLLM: the open-source LLM proxy (https://tokoscope.com/articles/litellm): Self-hosted routing vs OpenRouter, data privacy tradeoffs, and when the operational overhead is worth it.
- LangChain: the hidden token cost of abstraction (https://tokoscope.com/articles/langchain): How LangChain's templates, memory, and agent scratchpads add token overhead (measured up to 340%), and how to fix it.
- NotebookLM explained (https://tokoscope.com/articles/notebooklm): How Google's RAG-based research assistant works and what it teaches about cost-efficient AI product design.

### LLM Infrastructure & Operations
- LLM inference explained (https://tokoscope.com/articles/llm-inference): How prefill, decoding, and the KV cache work. Output tokens cost 3-5x more than input tokens because they're generated sequentially. Covers the main causes of inference cost spikes: verbose outputs, stuffed context windows, agentic tool calls, and no caching.
- LLM observability: what it is and why AI teams need it (https://tokoscope.com/articles/llm-observability): What to track beyond standard APM (token usage, cost attribution, latency, prompt quality, cache hit rate). Compares Tokoscope, LangSmith, Helicone, Braintrust, and Arize Phoenix. Includes a 2-minute SDK integration guide.
- RAG explained: how it works and what it costs (https://tokoscope.com/articles/rag-llm): Full RAG architecture (indexing + retrieval + generation). Compares RAG vs fine-tuning. Quantifies the token cost of context injection (2,000 extra tokens per call at 5 chunks = $1,000/day at scale). Covers chunking, reranking, and semantic caching to reduce RAG overhead.
- Fine-tuning LLMs: when it's worth it (https://tokoscope.com/articles/fine-tune-llm): The decision tree from prompt engineering to few-shot to RAG to fine-tuning. How OpenAI fine-tuning and LoRA work. Key insight: fine-tuning ROI often comes from inference savings (replacing a 1,500-token system prompt with 50 tokens).
- Qwen in 2026: Alibaba's LLM family explained (https://tokoscope.com/articles/qwen): Qwen2.5, Qwen2.5-Coder, Qwen2.5-Math, QwQ-32B, and Qwen-VL. Benchmark comparisons vs Llama 3.3-70B and Mistral-Large. Ollama pull commands and DashScope API pricing.
- Groq: the inference chip that makes LLMs 10x faster (https://tokoscope.com/articles/groq): How Groq's LPU achieves 500+ tokens/second via on-chip SRAM and deterministic execution. Pricing comparison for hosted models. When speed matters (real-time voice, live coding, agentic loops) vs when it doesn't.
- AnythingLLM: the open-source local RAG app (https://tokoscope.com/articles/anythingllm): Document ingestion, vector storage, multi-LLM support, and workspace management. Docker and desktop install. When to use it (internal tooling) vs building a custom RAG pipeline (customer-facing products).
- OpenCode: the open-source AI coding agent (https://tokoscope.com/articles/opencode): Terminal-based coding agent with multi-provider support. Compared to Claude Code, Aider, Cursor, and GitHub Copilot. Quantifies agentic coding token costs (10,000-200,000 tokens per coding session).

### Models, Rankings & Security
- SLM vs LLM (https://tokoscope.com/articles/slm-vs-llm): Small language models (under 10B parameters) vs large ones. Comparison table on speed, cost, hardware, and capability. Hybrid routing pattern: use SLMs for classification/extraction, LLMs for complex reasoning.
- Best open-source LLM for coding in 2026 (https://tokoscope.com/articles/best-open-source-llm-coding): Qwen2.5-Coder-32B (best overall, 65.9% HumanEval), DeepSeek-Coder-V2-16B (algorithms), Qwen2.5-Coder-7B (speed), StarCoder2-15B (niche languages), CodeLlama-70B (ecosystem). Full VRAM and benchmark table.
- LLM evals (https://tokoscope.com/articles/llm-evals): Three eval types — deterministic (rule-based), model-based (LLM-as-judge), human. Pairwise comparison vs absolute scoring. Tool comparison: Braintrust, LangSmith, Promptfoo, RAGAS, DeepEval, Inspect AI.
- LLM SEO (https://tokoscope.com/articles/llm-seo): Optimizing for AI search citation. llms.txt format and placement. JSON-LD Article schema. Content strategies: direct answers, comparison tables, specific numbers, semantic relevance over keyword density.
- LLM-as-a-judge (https://tokoscope.com/articles/llm-as-a-judge): Using one model to evaluate another. Biases: verbosity (longer = higher scores), self-preference (same model family), position (first shown gets advantage). Cost at scale: $350/day for 100K evals with GPT-4o. Mitigation: 1-5% sampling, cheaper judge for triage.
- vLLM (https://tokoscope.com/articles/vllm): PagedAttention borrows virtual memory paging for KV cache — reduces fragmentation from 60-80% to under 4%, enabling 2-24x higher throughput. Comparison to Ollama, llama.cpp, TGI, TensorRT-LLM. OpenAI-compatible endpoint.
- OWASP LLM Top 10 (https://tokoscope.com/articles/owasp-llm-top-10): All 10 risks: prompt injection, insecure output handling, training data poisoning, model DoS, supply chain, sensitive info disclosure, insecure plugin design, excessive agency, overreliance, model theft. Mitigations for each.
- MCP (https://tokoscope.com/articles/mcp): Model Context Protocol by Anthropic. Three primitives: resources (data), tools (functions), prompts (templates). Transport: stdio or HTTP/SSE. Adopted by Claude Desktop, Cursor, Zed, OpenAI, Google. Comparison to function calling.
- LLM optimization (https://tokoscope.com/articles/llm-optimization): 8 techniques ranked by ROI: semantic caching (20-40% savings), prompt compression (30-60%), right-sizing (10-30x), output constraints (20-40%), conversation trimming (40-80%), batch API (50%), quantization (50-75%), cost-per-endpoint tracking.
- LLM rankings (https://tokoscope.com/articles/llm-ranking): LMSYS Chatbot Arena (ELO from human votes), HF Open LLM Leaderboard (automated benchmarks, open-source only), Scale HELM (42 scenarios, 59 metrics), LiveBench (contamination-resistant, monthly refresh). Current top models by category.
- LLM suite (https://tokoscope.com/articles/llm-suite): Enterprise (M365 Copilot $30/user/month, Google Workspace AI, Salesforce Einstein), developer (Anthropic, OpenAI, Google AI Studio), open-source (LangChain ecosystem, Hugging Face, Ollama). Evaluation framework.
- Gemma 4 (https://tokoscope.com/articles/gemma-4): Sizes: 1B, 4B, 12B, 27B. Multimodal on 12B+, 128K context on all except 1B. Distilled from Gemini. Benchmark comparison vs Llama 3.2, Mistral, Qwen2.5, Phi-4. Gemma Terms of Use (not Apache — restrictions on competing products).

### Conversational AI & Model Guides
- Chat LLM: building conversational AI (https://tokoscope.com/articles/chat-llm): How multi-turn conversation works (LLMs are stateless — full history sent every turn). Three architectures: full history, sliding window, summarization. Token accumulation math: turn 50 costs 50x more than turn 1. Model recommendations by chat use case.
- LLM architecture explained (https://tokoscope.com/articles/llm-architecture): Tokenization, embeddings, transformer layers (attention + feed-forward), output projection, autoregressive decoding. Why attention is O(n²) with context length. Dense vs MoE vs state space models. How architecture drives inference cost.
- Llama: Meta's open-source LLM (https://tokoscope.com/articles/llama-llm): Timeline from Llama 1 (2023) to Llama 3.3 (2024). Size options: 1B, 3B, 8B, 70B. MMLU 86% at 70B. API options via Groq, Together AI, Replicate. Comparison vs GPT-4o and Claude on benchmark and cost.
- Meta's LLM strategy (https://tokoscope.com/articles/meta-llm): Why Meta open-sources Llama (commoditize the complement, talent, regulatory strategy). Meta's internal AI products (Meta AI assistant, ad targeting, content moderation). Open vs closed model comparison table.
- LLM stats: production benchmarks (https://tokoscope.com/articles/llm-stats): Average token counts by use case (Q&A: 250-700, summarization: 2,200-8,500, agentic: 2,500-12,000). Cache hit rates by type (FAQ: 40-70%, code: 5-15%). Cost per 1,000 API calls table. Compound optimization savings.
- Nous Hermes (https://tokoscope.com/articles/hermes): Fine-tuned Llama/Mistral variants by Nous Research. Strengths: instruction following, low refusals, structured output, persona consistency. Model lineup: Hermes 3 8B/70B/405B on Llama 3.1, Nous Hermes Mixtral. When to use vs vanilla Llama.
- Claude Opus (https://tokoscope.com/articles/claude-opus): Anthropic's frontier model. Pricing: $15/1M input, $75/1M output (5x Sonnet, 60x Haiku). Best for: complex reasoning, agentic tasks (leads SWE-bench), long-context synthesis. Routing strategy: default Sonnet, escalate to Opus.
- Google AI Studio (https://tokoscope.com/articles/google-ai-studio): Free Gemini API access — no billing required. Free tier: 10 RPM / 250K TPM on Gemini 2.5 Flash. Features: prompt UI, code export, fine-tuning, grounding. Comparison to Vertex AI (production). Works with Tokoscope SDK.

### July 26, 2026
- MiniMax (https://tokoscope.com/articles/minimax): Chinese AI lab, backed by Alibaba and Tencent. MiniMax-Text-01: 456B MoE, Lightning Attention (linear O(n) for long context), 1M token context, $0.20/1M input, $1.10/1M output, non-commercial license. MiniMax-VL-01: vision variant. Hailuo AI: text/image-to-video up to 1080p. OpenAI-compatible API at api.minimax.chat/v1.
- Kimi K3 pricing (https://tokoscope.com/articles/kimi-k3): Standard ~$0.60/$2.50 per 1M, thinking ~$0.60/$4.00, turbo ~$0.15/$0.60.
- DeepSeek API (https://tokoscope.com/articles/deepseek): deepseek-v4-flash (standard), deepseek-v4-pro. deepseek-chat and deepseek-reasoner retired July 24 2026.

### July 25, 2026 — Breaking
- Claude Opus 5 (https://tokoscope.com/articles/claude-opus-5): Released July 24 2026. Anthropic's 4th Claude 5 model in 2 months. Outperforms Fable 5 on 8/13 benchmarks at half the cost. Pricing: $5/1M input, $25/1M output (same as Opus 4.8). Key features: effort dial (low/med/high — reduces tokens at low effort), self-verification (recovers from errors without prompting), mid-task model switching. State-of-the-art on Frontier-Bench and GDPval-AA for coding/knowledge work. Behind Mythos 5 on cybersecurity. Default model: Claude Max. Model string: claude-opus-5-20260724.
- DeepSWE (https://tokoscope.com/articles/deepswe): Coding agent benchmark by Datacurve AI (May 2026). 113 original tasks (not GitHub issues), 91 repos, TypeScript/Go/Python/JS/Rust, program-based verifiers. Fixes SWE-bench contamination and over-specification. Prompts: behavior-focused, short, no interface blocks. Leaderboard (June 2026): GPT-5.5 67%/$7.23/task, Opus 4.8 59%, Sonnet 4.6 30%, Kimi K2.7 Code 31%, Gemini 3.1 Pro 12%/$9.48, DeepSeek V4 Pro 8%. DeepSeek and Mistral not yet evaluated. Scores 20-40pts lower than SWE-bench — the contamination gap.

### Fine-Tuning & LLM Development
- Unsloth (https://tokoscope.com/articles/unsloth): Fine-tuning acceleration library. Custom CUDA kernels (attention, RoPE, cross-entropy), memory-efficient backprop (chunked gradients), intelligent weight offloading. 2x faster, 60% less VRAM vs standard PEFT/LoRA. Hardware table: 8B fits RTX 3060 12GB, 70B fits 1x A100. GGUF export for Ollama/llama.cpp. Apache 2.0 (core), AGPL (some extensions).
- Build LLM from scratch (https://tokoscope.com/articles/build-llm-from-scratch): Decoder-only transformer components: embeddings, RoPE positional encoding, transformer blocks, MHA, FFN, RMSNorm, unembedding. BPE tokenizer. Data pipeline: crawl, deduplicate, quality-filter, mix domains. Training: AdamW + cosine LR. Cost table: 1B/2T tokens = ~$6K, 7B = ~$100K, 70B = ~$1.2M, frontier = $50M-$500M+. When justified: domain-specific corpus, compliance, IP control.
- LLM development (https://tokoscope.com/articles/llm-development): 7 stages. 1) Prototype (validate feasibility). 2) Prompt engineering (few-shot, CoT, format spec, compression). 3) RAG (chunk, embed, retrieve, inject). 4) Evaluation (golden dataset 50-200 examples, LLM-as-judge, task metrics, user feedback). 5) Cost optimization table (cheaper model 50-90%, compression 20-40%, caching 30-70%, fine-tuning 60-90%, batch API 50%). 6) Observability (cost/request, token ratio, latency, errors, quality, cache rate). 7) Security (input validation, output filtering, rate limiting, budget alerts).

### July 2026 — New Model Releases & Training Techniques
- Gemini 3.6 Flash (https://tokoscope.com/articles/gemini-3-6-flash): Released July 21 2026. Reasoning model with configurable effort. 65% fewer output tokens vs 3.5 Flash in some uses, 17% fewer overall. $1.50/1M input, $7.50/1M output. 1M context. OSWorld 83% (vs 78.4%). GDPval-AA 1421 (vs 1349). Knowledge cutoff March 2026. Available: AI Studio, Vertex, GitHub Copilot, Gemini app. Also released: 3.5 Flash-Lite ($0.30/$2.50, for Search/high-throughput), 3.5 Flash Cyber (security, limited access).
- DeepSeek R1 (https://tokoscope.com/articles/deepseek-r1): Released Jan 2025, MIT license. 671B total / 37B active MoE. Trained with pure RL (R1-Zero first, then cold-start SFT + RL for R1). AIME 2024: 79.8%, MATH-500: 97.3%, Codeforces: 2029 rating, GPQA Diamond: 71.5%, SWE-bench: 49.2%. Distilled dense models: Qwen-1.5B/7B/14B/32B and Llama-8B/70B. Now superseded by V4 Flash for general API use (deepseek-reasoner endpoint mapped to V4 Flash, retiring July 24 2026).
- DPO / Direct Preference Optimization (https://tokoscope.com/articles/dpo-llm): Alignment technique replacing RLHF. Eliminates reward model and RL via closed-form policy optimization on preference pairs. Training format: prompt + chosen response + rejected response. vs RLHF: no reward model, standard supervised loss, more stable, ~2x base SFT cost. Variants: IPO (overfit prevention), KTO (unpaired data), SimPO (no reference model), ORPO (single-pass SFT+alignment). Less verbose outputs than RLHF = lower output token cost.
- How to train an LLM (https://tokoscope.com/articles/how-to-train-llm): Three stages: 1) Pre-training (next-token prediction on 1-15T tokens, $10M-$500M+, thousands of GPUs). 2) SFT (instruction-response pairs, hours, $100-$50K). 3) Alignment (DPO or RLHF on preference pairs). Practical approaches: LoRA (adapter matrices, 10-100x cheaper than full FT), QLoRA (quantized base + LoRA, runs on RTX 4090). Cost table: 8B LoRA on A100 ~$20-50, 70B QLoRA ~$100-200. When to train vs RAG table.

### New Model Releases (July 2026)
- DeepSeek V4 Flash (https://tokoscope.com/articles/deepseek-v4-flash): Released April 24 2026, MIT license. 284B total / 13B active MoE params. 1M context, 384K max output. $0.14/1M input, $0.28/1M output. Architecture: DeepSeek Sparse Attention (DSA) hybrid reduces KV-cache 73%, Manifold-Constrained Hyper-Connections, Muon optimizer. AIME 2026: 96.7% (2nd of 14). Self-host on 1-2x A100 80GB. Note: deepseek-chat and deepseek-reasoner endpoints retire July 24 2026 — migrate to deepseek-v4-flash.
- OpenCode Go (https://tokoscope.com/articles/opencode-go): $5 first month, $10/month subscription for curated open coding model access via OpenCode. Not the Go programming language. OpenCode itself: 160K GitHub stars, 7.5M monthly users, built in Go, LSP integration for real-time compiler feedback. Go plan: single API key, OpenAI-compatible endpoints, beta pricing. Breaks even vs direct APIs at ~70M tokens/month on Gemini Flash or ~25M tokens on Qwen.

### LLM Security & Context Management
- LLM temperature (https://tokoscope.com/articles/llm-temperature): Temperature scales logits before softmax sampling. Low (0.0–0.3): deterministic, best for code/classification/factual QA. High (1.0–1.5): creative, varied, more verbose. Settings table by use case. Temperature vs top_p comparison. Higher temperature = longer outputs = higher token cost.
- LLM context window (https://tokoscope.com/articles/llm-context-window): Context window includes system prompt, conversation history, RAG docs, tool results, and user message. Sizes: GPT-4o 128K, Claude 200K, Gemini 1M. Lost-in-the-middle problem — models attend more to start/end of context. Attention scales quadratically. Cost: Gemini 2.5 Pro charges 2x over 200K tokens. Strategies: rolling window for chat history, system prompt compression, smaller RAG chunks, prompt caching (Anthropic/Google).
- LLM security (https://tokoscope.com/articles/llm-security): Four attacker categories: direct users, indirect injectors, external APIs, internal actors. Critical threats: prompt injection (direct/indirect), data leakage, excessive agent agency, model DoS, training data extraction. Security tools: Guardrails AI, NeMo Guardrails, Llama Guard, Promptfoo, LakeraGuard. 8-item production security checklist: input validation, output sanitization, access controls, rate limiting, secrets management, audit logging, budget alerts, red-team testing.

### LLM Development Practice
- LLM tools 2026 (https://tokoscope.com/articles/llm-tools): Full stack map — APIs (OpenAI, Anthropic, Google, Mistral, Groq), local engines (Ollama, vLLM, llama.cpp), orchestration (LangChain, LangGraph, CrewAI, MCP), coding (Claude Code, Cline, Cursor), RAG (Pinecone, pgvector, AnythingLLM), observability (Tokoscope, Helicone, LangSmith), evals (Promptfoo, DeepEval, RAGAS), cost optimization (caching, compression, routing, batching).
- LLMOps (https://tokoscope.com/articles/llm-ops): LLMOps vs MLOps comparison table. Five core practices: prompt versioning, cost monitoring (cost per request, cache hit rate, waste score), output quality monitoring (evals, LLM-as-judge, human review), safety guardrails (input validation, output filtering, rate limiting, anomaly detection), model management (pin versions, test before upgrading, maintain fallbacks). Toolchain by category.
- llama.cpp (https://tokoscope.com/articles/llama-cpp): C++ LLM inference engine for CPU and GPU. GGUF quantization table: FP16 (140GB 70B, 100% quality) to Q3_K_M (30GB, ~90% quality). Build from source, server mode (OpenAI-compatible). vs Ollama (easier) vs vLLM (faster on NVIDIA). Best for: CPU-only, Apple Silicon, edge hardware.
- Cline (https://tokoscope.com/articles/cline): Open-source VS Code AI coding agent (Apache 2.0). Features: file system access, terminal execution, browser control, MCP integration, any LLM provider. vs Claude Code (Anthropic only, terminal) vs Copilot (inline, paid) vs Cursor (VS Code fork, paid). Token cost: 15,000-150,000 tokens per complex task.
- Free LLM APIs (https://tokoscope.com/articles/free-llm-api): Comparison table of free tiers. Best: Google AI Studio (ongoing, 15 RPM / 1M TPM on Gemini Flash, data used for training). Groq (ongoing, 30 RPM, 250+ tok/s). OpenAI/Anthropic ($5 credit, one-time, expires). Mistral (1 RPM, experimental). Hugging Face (unreliable). Local alternatives: Ollama, llama.cpp.
- LLM for coding (https://tokoscope.com/articles/llm-for-coding): Three modes: autocomplete (GitHub Copilot), chat (any frontier model), agentic (Claude Code, Cline, OpenCode). Best model per task: complex multi-file (Claude Sonnet), simple functions (GPT-4o-mini/Gemini Flash), local (Qwen2.5-Coder-32B), algorithms (o3/Claude Opus). Prompting strategies. Cost: 50K-500K tokens per agentic session, $5-20/day with frontier models.

### Local LLMs & Cloud Platforms
- Best local LLM 2026 (https://tokoscope.com/articles/best-local-llm): Full guide by VRAM tier. Under 8GB: Qwen2.5-7B, Phi-4, Gemma 4-4B. 8-20GB: Qwen2.5-14B, DeepSeek-Coder-V2-16B. 20-48GB: Qwen2.5-32B, Bonsai-27B, Gemma 4-27B. 40GB+: Llama 3.3-70B, Qwen2.5-72B. By use case: coding (Qwen2.5-Coder-32B), reasoning (Phi-4/Qwen2.5-72B), vision (Gemma 4-12B). Local vs cloud decision table.
- Azure LLM (https://tokoscope.com/articles/azure-llm): Azure OpenAI Service vs direct OpenAI API. Key differences: data residency (your Azure region), compliance (SOC 2, ISO 27001, HIPAA, FedRAMP), 99.9% SLA, VNet/Private Link. Models: GPT-4o, o1, o3, DALL-E 3, Whisper, embeddings. Azure AI Foundry also hosts Llama, DeepSeek, Qwen. Compatible with Tokoscope via AzureOpenAI client.

### Specialized & Compact Models
- Bonsai LLM / Bonsai 27B (https://tokoscope.com/articles/bonsai-llm): Efficient 27B model via pruning and distillation. ~82% MMLU, ~18GB VRAM. Benchmark vs Llama 3.3 70B (86% MMLU, 40GB VRAM), Gemma 4 27B, Qwen2.5 32B, Mistral 24B. Apache 2.0. Fits on single A100 40GB with quantization.
- Fable LLM (https://tokoscope.com/articles/fable-llm): Domain-specialized for interactive narrative AI. Showrunner product enables AI-generated episodic content. Strengths: character consistency, narrative tracking, dialogue quality. Token cost challenge: 1,700-6,000 tokens overhead per session from character context, world context, history summary. Comparison vs GPT-4o and Claude Sonnet on creative vs general tasks.

### Emerging Models & Tools (July 2026)
- Kimi K3 (https://tokoscope.com/articles/kimi-k3): Moonshot AI reasoning model. 128K context, strong on math/code/logic. Pricing: ~$0.60/1M input, ~$2.50/1M output. Benchmark comparison vs GPT-4o, Claude Sonnet, DeepSeek-V3. Data privacy note: API sends data to China-based servers.
- Perplexity AI (https://tokoscope.com/articles/perplexity): AI answer engine using real-time RAG. 100M+ MAU. Sonar API is OpenAI-compatible. Comparison vs ChatGPT Search and Google AI Overviews on citation style, source transparency, and API availability. Implications for LLM SEO.
- Mesh LLM (https://tokoscope.com/articles/mesh-llm): Multi-agent LLM architecture patterns: sequential pipeline, parallel fan-out, hierarchical manager/worker, peer mesh. Token cost table: 1x (single) to 15-30x (5-agent hierarchical). Optimization: cheap models for coordination, pass summaries not transcripts, per-step budgets.
- OpenClaw (https://tokoscope.com/articles/openclaw): Open-source agent framework. Low complexity, low token overhead, explicit control flow. Comparison table vs LangChain (high overhead), CrewAI (role-based), AutoGen (conversational), LangGraph (graph-based). Works with any OpenAI-compatible client including Tokoscope-wrapped clients.

## Links

- Homepage: https://tokoscope.com
- Dashboard: https://app.tokoscope.com
- Docs: https://tokoscope.com/docs
- Pricing: https://tokoscope.com/pricing
- Articles: https://tokoscope.com/articles
- npm: https://www.npmjs.com/package/tokoscope
- PyPI: https://pypi.org/project/tokoscope/
- GitHub: https://github.com/tokoscope/sdk
- Contact: hello@tokoscope.com