Member(s): Sanjana Ashok
Advisor(s): Dipankar Raychaudhuri
Project goal: Evaluating how different information processing methods work on small or low-power (resource-constrained) edge devices with a focus on agentic retrieval and LLM memory/context management.
Tasks:
- Create a pipeline that gathers data from public sources (APIs, sensors, news feeds)
- Implement 3-4 processing methods that summarize the data (rule-based filtering, ML-based extraction, LLM-based summarization, and agentic retrieval)
- Measure and compare performance of each processing method (latency, accuracy, computational cost, etc.) on the data
- Make the framework portable and reproducible for future experimentation
Deliverable: Open-source benchmark suite that produces comparison report and gives concrete guidance on which information-processing method/context management strategy to utilize for a given type of data, task, or device budget.
- Agent harness — the software layer around an LLM that manages tools, memory, telemetry, and workflow orchestration. The harness (not the model alone) decides what actually reaches the model on each turn. This project compares two harnesses: LangGraph and OpenClaw.
- Agent memory types — working (current context window), semantic (general facts), procedural (skills/habits), and episodic (past events).
- Context-injection strategies — how much of a source document is placed into the prompt for a given question. Four are benchmarked:
- None — no document context; the model answers from its own knowledge (baseline/educated guess).
- Full — the entire document is injected.
- Selected — an LLM picks only the chunks relevant to the question and injects those.
- Compressed — the document is summarized, and the summary is injected.
- Context delivery modes — how memory reaches the model when the source is a large local record store rather than a single document. Four are measured:
- None — nothing injected; the baseline.
- Full dump — the entire store goes into the prompt.
- Bounded lookup — a local, deterministic search returns only the top matches, capped at a declared token budget.
- Oracle — the model is handed exactly the records that hold the answer, which separates a retrieval failure from an answer-model failure.
- Central research question: Which strategy keeps accuracy high while spending the fewest tokens?
Harnesses compared:
| Harness |
Notes |
| LangGraph |
Explicit, hand-controlled memory strategies; strong for isolating variables and reproducibility. |
| OpenClaw |
Open-source autonomous agent with its own memory system (workspace files + a memory_search tool); closer to how a real deployed agent behaves. |
Models evaluated (evolved over the course of the project)
- Initial:
Meta Llama 3.1 (local, via Ollama).
- Mid-project:
gpt-oss-20b (weak) and gpt-oss-120b (strong), via OpenRouter.
- Current:
qwen3.6-plus (weak) and qwen3.7-max (strong), served through an OpenCode Go endpoint.
Task set 1: document grounding — hand-written questions about domain-specific research papers with verified rubrics, in a balanced difficulty mix:
- True/False — factual claims; paper context needed to verify (easy).
- Multiple choice — specific values/methods from the paper (medium).
- Open-ended — explain a finding or mechanism; graded by rubric via an LLM judge (hard).
Task set 2: memory behavior — 24 questions about a local store of real Rutgers bus records, four per behavior, testing what happens when memory is large and contains records that are stale, conflicting, or merely similar to the answer:
- Single-record lookup — the answer sits in one record.
- Multi-record synthesis — the answer requires combining several records.
- Multi-time synthesis — the answer requires comparing versions of a record captured at different times.
- Stale versus current — the store holds a contradictory older record, and the model has to use the timestamps.
- Near-match rejection — a plausible look-alike record sits beside the real one.
- Absence / insufficient evidence — the correct answer is that the store does not contain it.
Grading
- Rule-based exact-match for True/False and multiple choice.
- LLM-as-judge with a rubric for open-ended answers.
Logging — every run is logged to a CSV (accuracy, latency, memory size, correctness) and, in the LangGraph pipeline, to Langfuse (token usage, temperature, traces), so results are fully auditable.
- Conducted a literature review on agentic memory systems and agent implementations (OpenClaw, Codex, LLM tool use, context windows), and compiled a research document.
- Set up the project GitHub repository and COSMOS Wiki page.
- First hands-on experiments with an OpenClaw implementation.
- Built a proof-of-concept LangGraph agentic workflow: a research-assistant agent that receives a request, calls simple tools, retrieves relevant memory, generates an answer, and logs the run.
- Agent workflow (LangGraph nodes):
Start → Memory Selection → Load Context → Response Generation → Log Run → End. The agent reads task and user-profile files, an LLM selects relevant memory, a file-reading tool loads the research source, the LLM builds the full prompt and answers, and the run is written to CSV.
- File architecture:
memory.json (user preferences, project facts, corrections), tasks.yaml (task definitions with a per-task memory_strategy), user_profile.md, and a research text file.
- Each run logged telemetry (selected memory IDs, estimated prompt tokens, latency) and enforced a structured output format (TL;DR, Key Points, and Why This Matters).
- Harness selection: compared LangGraph vs. AutoGen; chose LangGraph for being Python-native and the simplest framework for explicit state/context management.
- Literature review on evaluating LLM systems. Key takeaways: older benchmarks only test static recall and don't evaluate learning from feedback; longer context does not imply better memory; evaluation must go beyond accuracy to track latency, tokens, and storage; and no comprehensive agent benchmark yet exists. Reference works included MemoryBench and Memory for Autonomous LLM Agents: Mechanisms, Evaluation, and Emerging Frontiers.
- Built a more realistic data pipeline: pulled a year of New Brunswick weather data via Meteostat, cleaned it, and loaded it into a SQLite database.
- Defined an accuracy-measurement approach by generating ground-truth question–answer pairs over the data.
- Switched gears from the weather/SQLite pipeline to a paper-based QA benchmark, and formalized the four context-injection strategies (none/full/selected/compressed) as the core experimental axis.
- Created a complete pipeline: task → context strategy → model → answer → grading → logged result.
- Tested two models (
gpt-oss-20b weak, gpt-oss-120b strong) via OpenRouter with all four context strategies; implemented rule-based grading for T/F and MCQ and an LLM judge for open-ended; logging to CSV and Langfuse.
- Task set: 96 entries per run (2 models × 12 questions × 4 strategies), including a preference question type to test whether irrelevant context corrupts user-preference tracking.
- Preliminary results:
| Strategy |
Weak acc. |
Strong acc. |
Avg acc. |
Latency (s) |
| none |
0.652 |
0.833 |
0.745 |
5.23 |
| full |
0.917 |
0.917 |
0.917 |
8.87 |
| selected |
0.917 |
0.917 |
0.917 |
4.49 |
| compressed |
0.792 |
0.833 |
0.812 |
6.69 |
- Key finding: "selected" memory matched "full" accuracy at a fraction of the token cost and latency.
¶ Week 6 — Second harness + expanded evaluation
- Expanded the LangGraph benchmark to 2 papers (Needle Threading, MemoryBench), 9 questions/paper, a 3·3·3 T/F–MCQ–open-ended mix, and both
Qwen3.6-Plus / Qwen3.7-Max models: 144 entries per run.
- Model comparison: 69% (Qwen3.6-Plus) vs. 64% (Qwen3.7-Max) overall accuracy. The stronger model not winning flagged that parts of the task set may be too easy to separate the models.
- Efficiency result: selected memory held ~78% accuracy on ~1,000 prompt tokens versus ~35,900 for full context (roughly 97% fewer prompt tokens for comparable accuracy).
- Literature review: Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks? (Roberts, Han & Albanie, ICLR 2025): confirmed long-context memory degradation is real and that tokenizer differences matter when comparing token usage across model families.
- Began porting the benchmark onto OpenClaw. The first run was blocked by a harness bug: every condition returned identical answers, so no valid comparison could be drawn.
¶ Week 7 — OpenClaw harness fixed and first cross-harness results
- Diagnosed the bug: the first OpenClaw runs used a raw model call that sent only the question to the model and bypassed the system prompt, memory, and tools, so the paper never reached the model and every condition collapsed to the same output.
- Fix: routed each task through OpenClaw's agent runtime instead of a raw call, which delivered each paper through the agent's
MEMORY.md workspace file (OpenClaw injects into context automatically)
- Raised injection character cap: OpenClaw truncates files with more than 20K characters, and reference papers for the task set ranged from 67K-163K characters; raised the default to 400K characters.
- Verified injection: memory written per strategy climbs from 0 characters (none) to ~115K characters (full, averaged across papers)
- First cross-harness result (Qwen3.6-Plus; qwen3.7-max deferred due to a provider outage), 72 entries = 1 model × 2 papers × 9 questions × 4 strategies:
| Strategy |
OpenClaw |
LangGraph |
| Full |
100% |
100% |
| Selected |
100% |
78% |
| Compressed |
83% |
61% |
| None |
56% |
39% |
| Overall |
85% |
69% |
- Reading of the result: the harnesses agree perfectly at full context; OpenClaw scores higher on the reduced-context strategies, but two known confounds in this run (the chunk selector and the LLM judge both temporarily running on Qwen3.6) mean that gap is not yet attributable to the harness itself.
¶ Week 8 — Benchmark validity and interpretability
- Priority for the week: audit what each harness can actually reach during a run, and make every memory condition auditable rather than assumed.
- Found that the "no memory" condition was not a no-memory condition. With an empty
MEMORY.md, the OpenClaw agent used web_search and web_fetch across seven separate calls to pull the source paper off arXiv and answered from it. The agent had 24 tools available, 23 of which introduced confounds.
- Found that sessions carried context across tasks. Every task ran under one shared
--agent main session, so a task could see earlier tasks' history.
- Fixes applied:
| # |
Confound |
Fix |
| 1 |
Web access during the no-memory condition |
tools.web.search.enabled = false |
| 2 |
File read/exec tools available to the agent |
tools.profile = minimal |
| 3 |
Session history bleeding across tasks |
unique --session-key per run |
| 4 |
Model identity unverified |
log winnerModel and fallbackUsed per row |
| 5 |
Judge model from the same family as the models compared |
DeepSeek-V4-Flash, shared by both harnesses |
| 6 |
Selector model from the same family as the models compared |
DeepSeek-V4-Flash, shared by both harnesses |
| 7 |
System prompt size (17,305 chars for OpenClaw vs. 1,800 for LangGraph) |
not removable; documented and logged per row |
- End-to-end cost tracking. Context selection, answer generation, and the LLM judge each consume tokens and latency, so counting only the answer call understated the real cost of the selected and compressed runs. Cost is now logged per stage.
- Caching confound. OpenClaw caches the system prompt, so reading only the
input field hid 4,976 cached tokens per run and made its billed input tokens incomparable to LangGraph's for identical prompts.
- Reading of the result: the Week 7 finding that OpenClaw outperformed LangGraph on every reduced-context strategy is no longer trustworthy, since those runs had web access available and OpenClaw tied with LangGraph at full context. Week 7 results are retained for provenance but excluded from aggregation.
- Also identified: the paper task set measures document grounding rather than agent memory; the T/F–MCQ–open-ended tiers describe answer format rather than memory difficulty; and some tasks are answerable without the document, since T/F scored well above chance in the no-memory condition while open-ended collapsed.
¶ Week 9 — Scaling memory and building a place-bound context source
- Priority for the week: give the model a larger, messier memory to work with, and measure what actually reaches it.
- Built a versioned local context source (ContextStation). Rutgers bus data pulled from the PassioGo API (system 1268): 12 routes and 43 stops, which is 55 real records. A fetcher pulls the data and writes a snapshot, a dated frozen copy that never changes; the answering model reads only that pinned copy, so the correct answer to a task cannot shift between runs.
- A snapshot is published only after validation (no empty results, unique record IDs, coordinates inside New Jersey), and stores its provenance, capture time, record counts, and a SHA-256 hash. A failed refresh leaves the last valid version in place.
- Source limitations documented: Rutgers moved bus tracking to TripShot in July 2026, so
getVehicles() returns empty and live positions are no longer collectable; records carry no timestamps, so staleness cannot be derived from the source; the route id field is not unique (four routes share 10035), so records are keyed on myid; and the capture is reduced summer service, so a fall capture would hold more routes.
- Made memory scale an experimental variable. 55 records is roughly 6,000 tokens, at which size dumping everything and retrieving selectively produce nearly the same prompt. A seeded generator keeps all 55 real records and pads around them with labelled synthetic ones:
| Synthetic type |
What it contributes |
Share |
| Distractor |
A stop that does not exist, recombined from real street and campus names |
60% |
| Near-match |
"College Avenue Student Center North" sitting next to the real stop |
20% |
| Older version |
The same stop, backdated, with slightly different values |
12% |
| Stale conflict |
The same stop listing the wrong routes, with an older timestamp |
8% |
- The synthetic portion never duplicates the gold answer, since that would make retrieval easier as the store grows rather than harder. Every generated record carries
synthetic, synthetic_kind, derived_from, and superseded_by.
- A fixed seed reproduces the same records, IDs, ordering, and dataset hash, so any store can be rebuilt from three values: the seed, the scale level, and the hash of the snapshot it was built from.
- Separated storage scale from prompt scale. Records available in the store, records examined by retrieval, records selected, and tokens actually injected are four different numbers and are now logged separately.
- Results — 24 tasks × 4 delivery modes × 3 store sizes = 288 retrieval measurements, no API calls required:
| Store |
Records |
Full dump |
Bounded lookup |
Required records found |
Recall |
| Small |
250 |
17,197 tokens |
1,033 tokens |
16 of 18 tasks |
0.89 |
| Medium |
1,000 |
64,951 |
1,135 |
13 of 18 |
0.78 |
| Large |
5,000 |
330,793 |
1,166 |
11 of 18 |
0.61 |
- Recall is measured on the 18 tasks that require specific records; the other 6 are absence checks and no-memory controls, where there is nothing to retrieve.
- Key finding: the store grew 20× and full dump grew 19× with it, while bounded lookup barely moved and stayed inside its declared 1,200-token budget. But holding cost constant is not free: recall of the required records fell from 0.89 to 0.61, because the synthetic near-matches share keywords with the real stops and crowd the real records out of a fixed budget. Whether this is inherent to bounded retrieval or an artifact of the keyword-overlap scoring is still open.
- Designed 24 local-context tasks (see Benchmark Design above). Each task records its snapshot, query time, expected answer, the record IDs required to answer it, and any stale or distractor IDs planted to trip it.
- Gold answers are record IDs (
route:4056) rather than place names, since those IDs are not in any model's pretraining and the model cannot fall back on prior knowledge about Rutgers.
- The real data already contains near-miss pairs: Hill Center (NB) and Hill Center (SB) serve different route sets, as do the two Student Activities Centers, the two SoCam Apts stops, and the two Bergen buildings.
- 17 of the 24 expected answers are recomputed from the pinned snapshot automatically, so a hand-written mistake fails loudly.
- Closed more confounds:
| # |
Confound |
Fix |
| 1 |
The compressed condition returned empty context and was still graded correct |
the summarizer needed ~1,819 tokens against a 1,024 cap; raised to 4,096, and empty context now raises |
| 2 |
Cached tokens dropped from input cost (4,976 per run) |
cacheRead and cacheWrite summed into input, raw usage preserved |
| 3 |
Repeated runs of one condition reused the same session |
unique session key and run ID per invocation |
| 4 |
Each harness read its own independently edited copy of the shared files |
one canonical shared/ directory imported by both |
| 5 |
The two harnesses logged different result columns |
a single 52-column contract defined in shared/contract.py |
- Restructured the repository as the project record. Reorganized into
shared/, langgraph/, openclaw/, contextstation/, and results/, with a rewritten README covering setup, tests, snapshot refresh, and analysis; a pinned requirements.txt and a .env.example template; three test scripts that verify the structure, the pipeline end to end, and repeatability; snapshots and store manifests committed with their hashes; and the invalidated Week 7 and Week 8 runs preserved but labelled and excluded from aggregation.
- Context matters, and more isn't always better. Accuracy rises sharply from none to full, and the effect is largest on hard, open-ended questions (e.g., 100% with full memory vs. 33% with none in the OpenClaw run).
- Selected memory is the most efficient. It repeatedly matches full-memory accuracy while using a tiny fraction of the context (~1–2% of the characters; ~97% fewer prompt tokens).
- Bounded local lookup holds prompt cost flat as memory grows, but loses recall. Across a 20× increase in store size, full dump grew 19× while bounded lookup stayed inside its 1,200-token budget, but the share of required records it retrieved fell from 0.89 to 0.61 as near-match records accumulated.
- The Week 7 harness comparison is superseded. Those runs had web access available during the no-memory condition and shared sessions across tasks, so the 85% vs. 69% gap cannot be attributed to the harness. They are retained for provenance and excluded from aggregation. A valid cross-harness comparison has not yet been re-run.
- Language: Python
- Agent harnesses: LangGraph, OpenClaw (run on a remote Linux VM)
- Models & serving: Qwen3.6-Plus/Qwen3.7-Max via an Anthropic-compatible OpenCode Go endpoint; DeepSeek-V4-Flash as the shared selector and judge; earlier work used gpt-oss models via OpenRouter and Llama 3.1 via Ollama
- Data & evaluation: PassioGo API (Rutgers bus data), SQLite and Meteostat (earlier pipeline); pandas, PyYAML, tiktoken; Langfuse (tracing/telemetry)
- Repository layout:
shared/ — the single canonical copy both harnesses import: config.py, contract.py (the 52-column result schema), grader.py (rule-based + LLM-judge grading), memory_strats.py (the four context strategies), tasks.yaml (task/rubric definitions), and the two source papers as textlanggraph/ and openclaw/ — the runner, graph/agent definition, and model access specific to each harnesscontextstation/ — fetch_snapshot.py (refresh utility), snapshot_reader.py (pinned reader, no network), generate_store.py (scale generator), retrieve.py (the four delivery modes), verify_tasks.py (gold-answer verification), tasks_local.yaml (24 local-context tasks), plus committed snapshots and store manifestsresults/ — result CSVs and raw traces, with invalidated historical runs kept separately and labelled- Root —
README.md, a pinned requirements.txt, a .env.example template, and three test scripts (smoke_test.py, e2e_test.py, repeat_test.py)