Research brief 03 · Honest strengths + weaknesses · Source: 03-llm-agent-test.md
What we tested
Prefilled request (from test-location.md): 9-day Kyoto + Tokyo/Hakone for 2, mid-range, culture + food + calm pace, avoid crowds, private onsen once, tea ceremony, Nara/Uji day trip. Day-by-day plan, hotels, transport, bookings, budget, pitfalls.
Demo app (app.py) sends this to Muse Spark 1.3 — or uses curated fallback without an API key. See the frozen output: Agent Portal Demo.
What the LLM does well TODAY
- Fast, structured draft: full 9-day plan in ~20 seconds. Great starting skeleton.
- Logistics summary: Shinkansen, Hakone loop order, luggage forwarding — mostly correct, well explained.
- Food + etiquette lists: solid general advice (onsen rules, cash, reservations).
- Formatting: clean JSON → cards, timelines, budgets. Perfect for our portal UI.
- Follow-ups: rewrites quickly ("swap hotel for machiya," "add rest afternoon") without complaint.
Genuinely useful for a beginner agent: 70% of the research in seconds.
Where it falls short vs. a seasoned human
- No live truth: doesn't know what's sold out or price-changed today. Hallucinates times/prices if pushed.
- Generic picks: "top 10" hotels/temples, not the quiet machiya or the guide who's great with kids.
- No relationships: can't call the ryokan, secure the upgrade, or fix a cancellation at 9pm.
- Weak trade-offs: "both are great" instead of "skip Kinkaku-ji at noon, do it at open + add Uji."
- No accountability: if the plan fails, there's no one to call. Trust drops fast on high-spend trips.
- Shallow season nuance: knows "Golden Week is busy" but not the hour-by-hour temple strategy.
Scorecard (1–5)
| LLM (Muse Spark 1.3) | Seasoned agent | |
|---|---|---|
| Speed | 5 | 2 |
| Breadth of ideas | 5 | 3 |
| Accuracy (live prices/availability) | 2 | 5 |
| Personalization | 3 | 5 |
| Insider access / upgrades | 1 | 5 |
| Crisis help | 1 | 5 |
| Trust / accountability | 2 | 5 |
Takeaway — build for tomorrow, not just today
- Models will get better at facts and personalization. They won't get relationships, live inventory, or accountability on their own.
- LLM for draft + human + live supplier data for truth. AI drafts, agent curates, supplier feed confirms.
- Swap in better models anytime; keep the agent + supplier layer as the moat.