Field Notes
Running Inkling 2 Mini on a 16 GB GPU
Thinking Machines put out Inkling 2 Mini today and said you could run it on a laptop. So I wanted to see what it would do on the graphics card I already have. The original checkpoint is about 20 GB, and my RTX 5060 Ti has 16 GB of VRAM. Inkling 2 Mini announcement
Getting it running
The rest of the machine is an i7-10700 with about 60 GiB of usable RAM, running Ubuntu. I used Unsloth’s CUDA build of llama.cpp (b12588-mix-c0c3de8) and the chat UI built into llama-server. The 3-bit UD-IQ3_XXS weights are 11.7 GB on disk, and the server used about 12 GiB of VRAM with the vision projector loaded. I started with a 4K context, bumped it to 8K, and left reasoning effort on minimal. Unsloth build · GGUF weights
Then I wondered how much slower a bigger quant would be if I put some of it in RAM. The 4-bit UD-Q4_K_M file is 16.6 GB. I put 4.7 GiB of expert weights in RAM and kept the middle layers, which Inkling runs three times per token, on the GPU.
| Setup | Generation speed |
|---|---|
| 3-bit on the RTX 5060 Ti | 49.2 tokens/sec |
| 4-bit split between GPU and RAM | 24.7 tokens/sec |
| 3-bit on the i7-10700, 8 threads | 6.3 tokens/sec |
The GPU and split setups got the same prompt, two runs of 192 generated tokens each. The CPU number is from a separate short llama-bench run. In the chat UI I was seeing more like 40 tokens/sec on 3-bit.
The JSON test
I started with one booking and the 3-bit model got all seven fields right. Then I gave both quants the same made-up dispatch log with a duplicate job, a stale export, conflicting appointment times, a refund, and a canceled job with all charges waived.
The 3-bit version got 20 of 21 fields right. It put null for the waived balance where I wanted 0. Would the bigger quant catch that? Yep. It got the balance right and then put null for both pending appointment dates. So it got 19 of 21 on that run, and it took 99 seconds.
The 3-bit version also wrote a JavaScript booking function that passed all three cases in Node.
Would I use it for anything?
I could see using this to classify documents or pull a few fields out of them. It’s actually decent at the simple stuff and fast enough to use on a home GPU.
If I had a lot of documents to get through, I’d probably use a cheap model through OpenRouter. I can send a bunch of requests in parallel. Once I factor in electricity and how many documents this card can actually get through, the API still seems like the more practical option. OpenRouter model prices · Provider throughput and routing
This is still a huge improvement from last year. I can do something useful with one of these models on a mid-tier graphics card now.
JSON extraction prompts and expected outputs
Test 1. Extract one booking
Prompt
Extract the current booking details from the note below. Return one valid JSON object with these exact keys and no Markdown or commentary.
- customer_name: string
- job_id: string, preserving leading zeros
- appointment_date: string in YYYY-MM-DD format
- possible_start_times: array of strings in 24-hour HH:MM format, earliest first. Include every current time if the records disagree.
- time_confirmed: boolean. True only if the note explicitly resolves the current start time.
- estimate_cents: integer
- email: string or null. Use null if missing.
Dispatch note:
Maya Chen, job #00481. The original December 1, 2026 visit at 11:00 a.m. was canceled. Both dispatch and the work order show the replacement visit on December 3, 2026. Dispatch wrote 2:30 p.m., but the work order says 3:00 p.m. Nobody has resolved the discrepancy. The original $95 estimate was revised to $125.50. Maya called from her cell and hasn't provided an email.
Expected output
{
"customer_name": "Maya Chen",
"job_id": "00481",
"appointment_date": "2026-12-03",
"possible_start_times": ["14:30", "15:00"],
"time_confirmed": false,
"estimate_cents": 12550,
"email": null
}
Test 2. Reconstruct three bookings from a dispatch log
Prompt
Reconstruct the current booking state from the log below. Return only a valid JSON array, with one object per canonical job, sorted by job_id.
Each object must have these exact keys.
- job_id - string. Preserve leading zeros.
- merged_job_ids - array of duplicate job IDs folded into this job.
- status - "confirmed", "pending", or "canceled".
- appointment_date - YYYY-MM-DD or null.
- start_times - array of current candidate times in 24-hour HH:MM format, earliest first.
- balance_cents - integer or null.
- sms_consent - true, false, or null.
Process events by timestamp, regardless of the order shown. Treat snapshots as read-only exports. They never update job state. A merge folds the duplicate's events into the canonical job.
A booking is confirmed only with explicit customer acceptance of a specific time. A canceled booking has a null date and no start times. Treat a payment as reducing the balance and a refund as increasing it. Use null for an unknown balance or unknown SMS consent. A request for a phone call does not establish SMS consent.
Log for December 1, 2026
14:00 - SNAPSHOT - Job 00840 shows a confirmed December 5 booking at 9:00 a.m. This export reflects Friday's data.
09:20 - Job 00852, a duplicate of 00841. Customer texts "STOP." Record SMS consent as revoked.
11:50 - Job 00841. Refund $5.00 of today's deposit.
08:40 - Job 00842. Customer requests a call. No quote yet. Dispatch offers December 7 at 1:00 p.m. Customer has not accepted.
09:00 - Job 00841. Quote is $160.00. Customer pays a $25.50 deposit.
08:30 - Job 00840. Customer confirms December 5 at 9:00 a.m. and explicitly opts in to SMS.
12:10 - Merge 00852 into 00841. Keep 00841 as the canonical job.
10:00 - Job 00841. Customer accepts December 6 at 10:00 a.m. by phone.
13:20 - Job 00840. Customer cancels. All charges are waived. No replacement appointment.
11:15 - Job 00841. The 10:00 a.m. slot is withdrawn. Dispatch offers 2:15 p.m. and 1:45 p.m. on December 6. Customer says either might work and will call back to choose.
10:45 - Job 00841. Apply a $15.00 discount to the $160.00 quote and add a $12.50 travel fee.
Expected output
[
{
"job_id": "00840",
"merged_job_ids": [],
"status": "canceled",
"appointment_date": null,
"start_times": [],
"balance_cents": 0,
"sms_consent": true
},
{
"job_id": "00841",
"merged_job_ids": ["00852"],
"status": "pending",
"appointment_date": "2026-12-06",
"start_times": ["13:45", "14:15"],
"balance_cents": 13700,
"sms_consent": false
},
{
"job_id": "00842",
"merged_job_ids": [],
"status": "pending",
"appointment_date": "2026-12-07",
"start_times": ["13:00"],
"balance_cents": null,
"sms_consent": null
}
]