TanStack
Tutorials

Sign In Without the Model

Give the model a requestLogin tool with needsApproval: true. When a page asks for a password, the run stops and your chat shows a login card. The server fills the form in the browser, so the model never sees the password.

This tutorial uses React, TanStack Start, the harness, OpenAI, and a Docker desktop with Chromium. It takes about 30 minutes. Start Docker before you begin.

1. Create a Start app

shell
npx @tanstack/cli@latest create

Pick React. Then install the packages:

shell
npm i @tanstack/ai @tanstack/ai-react @tanstack/ai-openai @tanstack/ai-harness @tanstack/ai-persistence @tanstack/ai-computer-use @tanstack/ai-computer-sandbox @tanstack/ai-sandbox @tanstack/ai-sandbox-docker zod

Put your OpenAI key in .env.local:

shell
OPENAI_API_KEY=sk-...

Client and server

  • The desktop is a Docker container with a screen and Chromium. Each chat thread gets its own container.
  • The server runs the harness. The model clicks and types on the desktop with the computer tool. The requestLogin tool fills the sign-in form.
  • The client shows the chat and the login card. It sends the password to its own route, not to the chat.

The approval of a tool call goes into the harness log. So the card approves the call with only "yes", and the password takes another path.

2. Build the desktop

Create a sandbox folder in your app with five files.

sandbox/Dockerfile installs the screen, the tools of the sandbox computer driver, Chromium, and Python:

dockerfile
FROM debian:bookworm-slim

RUN printf 'Acquire::Retries "10";\nAcquire::ForceIPv4 "true";\n' > /etc/apt/apt.conf.d/80retries \
  && apt-get update \
  && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
    xvfb xdotool scrot x11-utils chromium fonts-dejavu-core \
    python3 python3-websockets ca-certificates \
  && rm -rf /var/lib/apt/lists/*

COPY policy.json /etc/chromium/policies/managed/policy.json
COPY desktop.sh browser.sh login.py /opt/desktop/
RUN chmod +x /opt/desktop/desktop.sh /opt/desktop/browser.sh

sandbox/policy.json keeps passwords off the screen. The model sees the screen and moves the mouse, so Chromium must not offer to show a saved password:

json
{
  "PasswordManagerEnabled": false,
  "AutofillAddressEnabled": false,
  "AutofillCreditCardEnabled": false,
  "URLBlocklist": ["javascript://*"],
  "RestoreOnStartup": 1
}

RestoreOnStartup opens the last tabs when Chromium starts again. It also keeps the session cookies, so a restarted Chromium is still signed in.

sandbox/desktop.sh is the main process of the container. It starts the screen and Chromium:

shell
#!/bin/sh
rm -f /tmp/.X0-lock /tmp/chromium-paused /tmp/chromium-stopped
Xvfb :0 -ac -screen 0 1024x768x24 -nolisten tcp &
until DISPLAY=:0 xdpyinfo >/dev/null 2>&1; do sleep 0.2; done

while true; do
  # browser.sh stop pauses this loop while a snapshot saves or restores the
  # profile. The loop tells browser.sh that Chromium is stopped.
  if [ -e /tmp/chromium-paused ]; then
    touch /tmp/chromium-stopped
    while [ -e /tmp/chromium-paused ]; do sleep 0.2; done
    rm -f /tmp/chromium-stopped
  fi
  # --no-sandbox: Chromium runs as root in the container, which is the sandbox.
  DISPLAY=:0 chromium --no-sandbox --test-type --no-first-run \
    --no-default-browser-check --disable-dev-shm-usage \
    --user-data-dir=/root/chromium-profile --password-store=basic \
    --remote-debugging-port=9222 \
    --window-position=0,0 --window-size=1024,768
  sleep 1
done
  • --user-data-dir keeps the cookies. A restarted container is still signed in.
  • --password-store=basic uses the same cookie key in every container. A profile that a new container restores can read its cookies.
  • --remote-debugging-port listens on 127.0.0.1 inside the container only. That port never leaves the container.

sandbox/browser.sh stops and starts Chromium, so a snapshot saves a complete profile. Copy it from the example:

  • browser.sh stop pauses the loop in desktop.sh, then closes Chromium with the debugging command Browser.close. A clean close writes the cookies to disk. A kill signal can lose them.
  • browser.sh stop returns when the loop says that Chromium is stopped. Then no new Chromium can start while the profile saves.
  • browser.sh start ends the pause, then waits until Chromium answers on its debugging port.

sandbox/login.py signs in. Copy it from the example. It reads the site, the username, and the password from its environment, then talks to Chromium over the debugging port:

  1. It opens the site in the current tab.
  2. It stops if the page is not on the origin of the site, so a redirect cannot get the password.
  3. It fills the username field and the password field, then submits the form.
  4. It clears every password field, and prints the address of the page.

Build the image:

shell
docker build -t computer-use-login-desktop sandbox

3. Share the tool definition

The server and the browser both import the tool definition. Create src/request-login.ts:

ts
import { toolDefinition } from '@tanstack/ai'
import { z } from 'zod'

export const requestLoginTool = toolDefinition({
  name: 'requestLogin',
  description:
    'Ask the user to sign in to a site. The user types the username and password in a form that you never see, and the server fills the sign-in form in the browser. Call it when a page asks for a password. Pass the address of the sign-in page.',
  // The run stops until the user answers the login card.
  needsApproval: true,
  inputSchema: z.object({
    site: z
      .url({ protocol: /^https$/ })
      .describe(
        'The address of the sign-in page, for example https://example.com/login',
      ),
  }),
})

4. Set up the harness

Create src/harness.ts. It has the Docker provider, three stores, the tool, and the harness.

First the provider and the sandbox ids. sandboxIds stores the container of each thread. With it, the driver reconnects to the same container later, so the thread finds its signed-in browser again:

ts
import { mkdir, readFile, rename, writeFile } from 'node:fs/promises'
import { dirname, resolve } from 'node:path'
import { createHarnessHost, defineHarness } from '@tanstack/ai-harness'
import { permissions } from '@tanstack/ai-harness/plugins'
import { createHarnessHandler } from '@tanstack/ai-harness/server'
import { computerUse } from '@tanstack/ai-computer-use/harness'
import {
  saveComputerSnapshot,
  sandboxComputer,
} from '@tanstack/ai-computer-sandbox'
import { memorySandboxSnapshots } from '@tanstack/ai-sandbox'
import type { SandboxHandle } from '@tanstack/ai-sandbox'
import { dockerSandbox } from '@tanstack/ai-sandbox-docker'
import { openaiText } from '@tanstack/ai-openai'
import { memoryPersistence } from '@tanstack/ai-persistence'
import { z } from 'zod'
import { requestLoginTool } from './request-login'

const provider = dockerSandbox({
  image: 'computer-use-login-desktop',
  keepAliveCommand: ['/opt/desktop/desktop.sh'],
})

// The container id of each thread, in a file.
const SANDBOX_IDS_FILE = resolve('.data/sandbox-ids.json')
const sandboxIdsSchema = z.record(z.string(), z.string())
let sandboxIdsWrite: Promise<void> = Promise.resolve()

async function readSandboxIds() {
  try {
    return sandboxIdsSchema.parse(
      JSON.parse(await readFile(SANDBOX_IDS_FILE, 'utf8')),
    )
  } catch (error) {
    if (error instanceof Error && 'code' in error && error.code === 'ENOENT') {
      return {}
    }
    throw error
  }
}

const sandboxIds = {
  get: async (threadId: string) => (await readSandboxIds())[threadId],
  set: (threadId: string, sandboxId: string) => {
    // One write at a time, so two threads do not lose each other's id.
    const write = sandboxIdsWrite.then(async () => {
      const ids = await readSandboxIds()
      await mkdir(dirname(SANDBOX_IDS_FILE), { recursive: true })
      // Write a new file, then rename it, so a read never sees half a file.
      const temporary = `${SANDBOX_IDS_FILE}.${crypto.randomUUID()}`
      await writeFile(
        temporary,
        JSON.stringify({ ...ids, [threadId]: sandboxId }),
      )
      await rename(temporary, SANDBOX_IDS_FILE)
    })
    // A failed write fails this call only. The next write still runs.
    sandboxIdsWrite = write.catch(() => {})
    return write
  },
}

Then the snapshots of the Chromium profile. persist.reconnect brings back a container that still exists. A snapshot brings the profile into a new container, when the old one is gone. This also works with providers whose disk does not last, like Cloudflare:

ts
const PROFILE = '/root/chromium-profile'
// Only the files that keep a sign-in. Caches stay out of the snapshot.
const PROFILE_FILES = [
  'Local State',
  'Default/Cookies',
  'Default/Cookies-journal',
  'Default/Network/Cookies',
  'Default/Network/Cookies-journal',
  'Default/Preferences',
  'Default/Sessions',
  'Default/Local Storage',
  'Default/Session Storage',
]
const keepInProfile = (path: string) =>
  PROFILE_FILES.some(
    (file) =>
      path === file ||
      path.startsWith(`${file}/`) ||
      file.startsWith(`${path}/`),
  )

async function browser(sandbox: SandboxHandle, action: 'stop' | 'start') {
  const result = await sandbox.process.exec(`/opt/desktop/browser.sh ${action}`)
  if (result.exitCode !== 0) {
    throw new Error(`Could not ${action} Chromium: ${result.stderr.trim()}`)
  }
}

const snapshots = {
  store: await memorySandboxSnapshots(),
  root: PROFILE,
  policy: { exclude: (path: string) => !keepInProfile(path) },
  // Chromium writes its profile all the time. Stop it, so the files are complete.
  before: (sandbox: SandboxHandle) => browser(sandbox, 'stop'),
  after: (sandbox: SandboxHandle) => browser(sandbox, 'start'),
}
  • The driver saves the profile when a thread closes or goes idle. The tool below also saves it right after a sign-in.
  • memorySandboxSnapshots() keeps the snapshots in memory, like the chat. For a user who comes back the next day, build the stores with createSandboxSnapshots on your database.
  • The profile holds session cookies. Keep the blob store secret, and encrypt it in production.

Then the logins. The card sends a login for one requestLogin call, and the tool takes it once. A login that nobody takes is deleted after 5 minutes:

ts
const LOGIN_TTL_MS = 5 * 60_000
const logins = new Map<
  string,
  {
    site: string
    username: string
    password: string
    timer: ReturnType<typeof setTimeout>
  }
>()

export const loginSchema = z.object({
  toolCallId: z.string().min(1).max(200),
  site: z.url({ protocol: /^https$/ }).max(2000),
  username: z.string().min(1).max(500),
  password: z.string().min(1).max(500),
})

export function saveLogin({
  toolCallId,
  ...login
}: z.infer<typeof loginSchema>) {
  clearTimeout(logins.get(toolCallId)?.timer)
  const timer = setTimeout(() => logins.delete(toolCallId), LOGIN_TTL_MS)
  timer.unref()
  logins.set(toolCallId, { ...login, timer })
}

function takeLogin(toolCallId: string, site: string) {
  const login = logins.get(toolCallId)
  logins.delete(toolCallId)
  clearTimeout(login?.timer)
  if (!login) {
    throw new Error(
      'The user gave no username and password. Call requestLogin again.',
    )
  }
  if (new URL(login.site).origin !== new URL(site).origin) {
    throw new Error(
      `The user signed in for ${new URL(login.site).origin}, not ${new URL(site).origin}.`,
    )
  }
  return login
}

Then the tool. It runs after the user approves the card:

ts
// No query and no fragment: they can hold form values.
const pageAddress = (url: string) => {
  const { origin, pathname } = new URL(url)
  return origin + pathname
}

const loginResultSchema = z.object({
  url: z.string(),
  hasPasswordField: z.boolean(),
})

const requestLogin = requestLoginTool.server(async ({ site }, context) => {
  const threadId = context?.threadId
  const toolCallId = context?.toolCallId
  if (!threadId || !toolCallId) {
    throw new Error('requestLogin runs only in a chat thread.')
  }
  const login = takeLogin(toolCallId, site)

  // The user can answer the card hours later. `resume` starts a stopped
  // container again.
  const id = await sandboxIds.get(threadId)
  const sandbox = id === undefined ? null : await provider.resume({ id })
  if (!sandbox) {
    throw new Error(
      'This thread has no computer. Take a screenshot, then call requestLogin again.',
    )
  }

  // The password goes in the environment of one process, not on its command line.
  const result = await sandbox.process.exec('python3 /opt/desktop/login.py', {
    env: {
      // The normal form of the address, so its origin matches Chromium's.
      LOGIN_SITE: new URL(site).href,
      LOGIN_USERNAME: login.username,
      LOGIN_PASSWORD: login.password,
    },
  })
  if (result.exitCode !== 0) {
    throw new Error(`The sign-in failed: ${result.stderr.trim()}`)
  }
  const page = loginResultSchema.parse(JSON.parse(result.stdout))
  // A wrong password. The model calls requestLogin again, and the user retries.
  if (page.hasPasswordField) {
    throw new Error(
      `The sign-in failed: ${pageAddress(page.url)} still asks for a password. Call requestLogin again, so the user can try again.`,
    )
  }
  // Save the signed-in profile now. A server that stops skips the idle save.
  await saveComputerSnapshot({ sandbox, threadId, snapshots })
  return `Signed in. The browser shows ${pageAddress(page.url)}. Take a screenshot to see it.`
})
  • takeLogin deletes the login, so the server uses each password once.
  • The tool result has no password. The model gets only the address of the page, with no query and no fragment.

Last, the harness and its HTTP handler:

ts
const computer = computerUse({
  driver: sandboxComputer({
    provider,
    persist: { reconnect: sandboxIds, snapshots },
  }),
  screenshotDelay: 1000,
})

const agent = defineHarness({
  name: 'computer-use-login',
  adapter: openaiText('gpt-6.1-sol'),
  systemPrompts: [
    'You use a Linux desktop with Chromium on it. When a page asks you to sign in, call requestLogin with the address of the sign-in page. Never ask for a password in the chat, and never type one.',
    'When requestLogin says that the page still asks for a password, call requestLogin again. When it fails for another reason, do not call it again: tell the user the error.',
  ],
  tools: [requestLogin],
  plugins: () => [permissions(), computer],
})

const host = createHarnessHost({ persistence: memoryPersistence() })

export const handler = createHarnessHandler({
  host,
  harness: agent,
  openAccess: true,
})
  • sandboxComputer makes one container for each thread. With persist.reconnect, it keeps the container and reconnects to it later. With persist.snapshots, a new container gets the saved profile.
  • memoryPersistence() keeps the chat in memory. A server restart loses the chat, but not the container. For a user who comes back the next day, use a persistence store in your database. See Chat persistence.
  • openAccess serves every request with no login. Use it only on your own machine. See Serve with no login.

The Docker provider loads dockerode, which has native addons. Keep it out of the bundle in vite.config.ts:

ts
export default defineConfig({
  optimizeDeps: { exclude: ['dockerode'] },
  ssr: { external: ['dockerode'] },
  build: { rollupOptions: { external: ['dockerode'] } },
  // ...your plugins
})

5. Add the two routes

Create src/routes/api.harness.$.ts. It serves the harness under /api/harness:

ts
import { createFileRoute } from '@tanstack/react-router'
import { handler } from '@/harness'

export const Route = createFileRoute('/api/harness/$')({
  server: {
    handlers: {
      GET: ({ request }) => handler(request),
      POST: ({ request }) => handler(request),
    },
  },
})

Create src/routes/api.login.ts. The card sends the login here. The route checks the input, keeps the login in memory, and logs nothing:

ts
import { createFileRoute } from '@tanstack/react-router'
import { loginSchema, saveLogin } from '@/harness'

export const Route = createFileRoute('/api/login')({
  server: {
    handlers: {
      POST: async ({ request }) => {
        const body: unknown = await request.json().catch(() => undefined)
        const login = loginSchema.safeParse(body)
        if (!login.success) {
          return Response.json(
            { error: 'Send a thread id, a site, a username, and a password.' },
            { status: 400 },
          )
        }
        saveLogin(login.data)
        return new Response(null, { status: 204 })
      },
    },
  },
})

6. Show the login card

In src/routes/index.tsx, give useChat the tool definition. The tool has no browser code, but useChat now types its approval:

tsx
import { fetchServerSentEvents, useChat } from '@tanstack/ai-react'
import { requestLoginTool } from '@/request-login'

const THREAD_ID = 'computer-use-login'

const { messages, sendMessage, interrupts } = useChat({
  connection: fetchServerSentEvents('/api/harness/run'),
  threadId: THREAD_ID,
  tools: [requestLoginTool.client()],
  // Load the thread from the server after a reload.
  persistence: true,
})

Add the card. It sends the login to /api/login first. Then it approves the tool call:

tsx
function LoginCard({
  toolCallId,
  site,
  onAnswer,
}: {
  toolCallId: string
  site: string
  onAnswer: (approved: boolean) => void
}) {
  const [username, setUsername] = useState('')
  const [password, setPassword] = useState('')
  const [error, setError] = useState<string>()

  const signIn = async (event: React.FormEvent) => {
    event.preventDefault()
    setError(undefined)
    try {
      const response = await fetch('/api/login', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ toolCallId, site, username, password }),
      })
      if (!response.ok) {
        throw new Error(`The server did not take the login (${response.status}).`)
      }
      setPassword('')
      onAnswer(true)
    } catch (sendError) {
      setError(sendError instanceof Error ? sendError.message : String(sendError))
    }
  }

  return (
    <form onSubmit={signIn}>
      <p>Sign in to {new URL(site).origin}</p>
      <input
        value={username}
        onChange={(event) => setUsername(event.target.value)}
        autoComplete="username"
        aria-label="Username"
        required
      />
      <input
        type="password"
        value={password}
        onChange={(event) => setPassword(event.target.value)}
        autoComplete="current-password"
        aria-label="Password"
        required
      />
      {error ? <p>{error}</p> : null}
      <button type="submit">Sign in</button>
      <button type="button" onClick={() => onAnswer(false)}>
        Cancel
      </button>
    </form>
  )
}

Show a card for each approval of requestLogin, below the messages:

tsx
{interrupts.map((interrupt) =>
  interrupt.kind === 'tool-approval' ? (
    <LoginCard
      key={interrupt.id}
      toolCallId={interrupt.toolCallId}
      site={interrupt.originalArgs.site}
      onAnswer={(approved) => interrupt.resolveInterrupt(approved)}
    />
  ) : null,
)}

The harness keeps each screenshot in its media store. A tool result has a harness-media:<id> URL for it. Show it from the media path of the handler:

tsx
const src = `/api/harness/media?${new URLSearchParams({ threadId: THREAD_ID, id })}`

The example has the full page, with the screenshots and the card for OpenAI safety checks.

7. Sign in

Start the app with npm run dev. Send this task:

plaintext
Open https://the-internet.herokuapp.com/login in Chromium and sign in.
When you are signed in, say what the page shows and stop.

The model opens the practice page and calls requestLogin. Type tomsmith and SuperSecretPassword! in the card, then click Sign in. The model says that the page shows the Secure Area. Your password is not in the chat, the tool result, or the harness log.

8. Watch the desktop and take over

Show the desktop live at the end of the chat, in place of the screenshots. When the model is stuck, it calls a handOff tool, and the user uses the desktop until they click Done.

  1. Add the live view to sandbox/Dockerfile. Put x11vnc novnc python3-websockify after ca-certificates in the apt-get install line. Then copy a viewer page that shows only the screen, with no noVNC controls. Get view.html from the example:

    dockerfile
    
    COPY view.html /usr/share/novnc/view.html
  2. Start it in sandbox/desktop.sh, after the until line. Each start makes a new VNC password in /dev/shm, which is empty after each container start:

    shell
    head -c 256 /dev/urandom | tr -dc 'A-Za-z0-9' | head -c 8 > /dev/shm/vnc-password
    x11vnc -display :0 -localhost -forever -shared -quiet \
      -passwdfile /dev/shm/vnc-password -rfbport 5900 &
    websockify --web /usr/share/novnc 6080 localhost:5900 >/dev/null 2>&1 &
  3. Build the image again, then publish the noVNC port in src/harness.ts. Docker publishes it on 127.0.0.1 only:

    ts
    const provider = dockerSandbox({
      image: 'computer-use-login-desktop',
      keepAliveCommand: ['/opt/desktop/desktop.sh'],
      publishPorts: [6080],
    })
  4. Add a function to src/harness.ts that gives the address of the live view, with its password:

    ts
    export async function liveViewUrl(threadId: string) {
      const id = await sandboxIds.get(threadId)
      const sandbox = id === undefined ? null : await provider.resume({ id })
      if (!sandbox) return null
      const [{ url }, password] = await Promise.all([
        sandbox.ports.connect(6080),
        // desktop.sh writes it after each start. Wait for it, up to 10 s.
        sandbox.process.exec(
          'for i in $(seq 50); do test -s /dev/shm/vnc-password && cat /dev/shm/vnc-password && exit 0; sleep 0.2; done; echo "desktop.sh wrote no password." >&2; exit 1',
        ),
      ])
      if (password.exitCode !== 0) {
        throw new Error(`No live view password: ${password.stderr.trim()}`)
      }
      const params = new URLSearchParams({ password: password.stdout.trim() })
      return `${url}/view.html?${params}`
    }

    Return it from a GET /api/live?threadId=... route as { url }. Copy api.live.ts from the example.

  5. Create src/hand-off.ts. The tool has needsApproval: true, so the run stops until the user clicks Done:

    ts
    import { toolDefinition } from '@tanstack/ai'
    import { z } from 'zod'
    
    export const handOffTool = toolDefinition({
      name: 'handOff',
      description:
        'Give the desktop to the user, and wait until the user is done. Call it when you are stuck and cannot continue alone: a captcha, a two-factor code, a page that you cannot use, or a choice that only the user can make. Do not call it for a sign-in: call requestLogin. Then take a screenshot to see what the user did.',
      needsApproval: true,
      inputSchema: z.object({
        reason: z
          .string()
          .describe('What the user must do, in one sentence. The user reads it.'),
      }),
    })
  6. Give the server tool to the harness in src/harness.ts:

    ts
    const handOff = handOffTool.server(
      () => 'The user is done. Take a screenshot to see the screen.',
    )
    
    // In defineHarness:
    tools: [requestLogin, handOff],
  7. In src/routes/index.tsx, add handOffTool.client() to the tools of useChat. Remove the screenshots, and show the live view in an <iframe> after the last message. Add &view_only=1 to its address, except while a handOff approval waits:

    tsx
    const handedOff = interrupts.some(
      (interrupt) =>
        interrupt.kind === 'tool-approval' && interrupt.toolName === 'handOff',
    )
    
    <iframe
      src={handedOff ? url : `${url}&view_only=1`}
      title="Live view of the desktop"
    />

    For the handOff approval, show interrupt.originalArgs.reason with a Done button that calls interrupt.resolveInterrupt(true). Copy the LiveView and HandOffCard components from the example.

Start the app, and send a task that needs a person, for example "Open https://www.google.com/recaptcha/api2/demo and send the form." The model calls handOff. Tick the captcha in the live view, then click Done. The model continues.

The model sees only the screenshots that it takes after you click Done. A password field that the site masks shows only dots. A site that shows the password as text shows it in the screenshot too. Hide the password before you click Done, or use the login card.

The full example is on GitHub: TanStack/ai examples/react/computer-use-login. To host it, see Run it on a server in its README.