Skip to content

Allow clone() to add/override initialPrompts #213

Description

@ibnesayeed

Use case

An app hosts multiple independent "conversations" that share a common setup but diverge in some static, per-conversation context. To avoid redundant model loads, it creates one shared base LanguageModel session and has each conversation work from its own session.clone(). Each clone needs its own distinct static context (e.g. a description of that conversation's available data) that persists reliably for the clone's lifetime, ideally with the same "never removed" guarantee initialPrompts' system message gets on the original session.

clone() doesn't currently support adding a new initialPrompts entry (or overriding the inherited one) at clone time.

What we tried

session.append([{ role: 'system', content: '...' }]) right after clone(), hoping the system role would carry the same persistence guarantee documented for a session's original initialPrompts system message ("the system prompt, which is never removed" during context-window trimming).

What we found

Empirically, it doesn't persist. We appended a distinctive marker via append(), tested with both system and user roles, then padded the session with ~30-60 filler prompt() calls to force normal context-window pressure. In both cases:

  • No error was ever thrown (no QuotaExceededError, nothing), the content was dropped silently.
  • Asking the model to recall the appended content afterward failed, it had no memory of it.

As a control, a genuine initialPrompts system message set at the original create() call survived the same stress test (60 padding turns) with correct recall intact. So the "never removed" guarantee appears to be specific to the literal message set via initialPrompts at a session's original creation, not to "any message with role: 'system'" in general, append() doesn't grant it, regardless of role.

Request

Could clone({ initialPrompts: [...] }) (or similar) let a clone add its own protected, never-evicted context on top of what it inherited from the parent? Right now there's no way to get both benefits at once: sharing one base session's model-load cost across clones, and giving each clone reliably persistent per-clone context. The only current path to real persistence is an independent create({ initialPrompts }) per conversation, which gives up the shared-base-session benefit entirely.

Minimal reproduction

async function testPersistence(seedFn, label) {
  const opts = { expectedInputs: [{ type: 'text', languages: ['en'] }], expectedOutputs: [{ type: 'text', languages: ['en'] }] };
  const session = await seedFn(opts);
  const recall = 'What color did I mention earlier? Answer in one word, or say "unknown".';

  console.log(`[${label}] immediate recall:`, await session.prompt(recall));

  // Force normal context-window pressure with filler turns.
  const filler = 'Reply with just the word "OK". '.repeat(20);
  for (let i = 0; i < 40; i++) {
    await session.prompt(`${filler} (turn ${i})`);
  }

  console.log(`[${label}] recall after 40 filler turns:`, await session.prompt(recall));
  session.destroy();
}

// A: append() with role "system", added to an already-created session.
await testPersistence(async (opts) => {
  const session = await LanguageModel.create(opts);
  await session.append([{ role: 'system', content: 'Remember this color: cerulean.' }]);
  return session;
}, 'append() role=system');

// B: initialPrompts system message, set at the session's original create() call.
await testPersistence(
  (opts) => LanguageModel.create({ ...opts, initialPrompts: [{ role: 'system', content: 'Remember this color: cerulean.' }] }),
  'initialPrompts role=system',
);
[append() role=system] immediate recall: Cerulean.
[append() role=system] recall after 40 filler turns: unknown
[initialPrompts role=system] immediate recall: Cerulean. 
[initialPrompts role=system] recall after 40 filler turns: Cerulean

Expected: both recall "cerulean" after the filler turns, since both were added with role: 'system'.

Actual: A forgets it (no error thrown at any point, the content is just silently gone), but B still recalls it correctly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions