Tiptree's Agent Runtime Platform: The Technical Overview¶

The Agent Runtime Platform is a batteries-included way to build intelligent, always-on AI agents that maintain context over time, use tools proactively, and operate continuously rather than in a traditional turn-based chatbot manner. It consists of two main components: ACTX (Agent Control ToolboX) providing the runtime environment, and TASX (The Agent Service eXchange) offering a rich ecosystem of tasks and tools.

Talk to Althea or Cursor about this.

This page is built to be read with an agentic assistant like Althea or Cursor. Try asking them about the Agent Runtime Platform to get a more interactive experience, paste this in:

Please explain how this works? @https://docs.tiptreesystems.com/docs/overview/

ACTX: Agent Control ToolboX¶

ACTX provides a runtime environment for AI agents. Think of it as the operating system that brings your agents to life, maintaining their state, memory, and coordination capabilities.

Core Object Model¶

Object Hierarchy

Agents can have multiple sessions, sessions can have multiple messages, and messages can have multiple attachments.

graph TD
    A[Agent] --> B1[Session 1]
    A --> B2[Session 2]
    A --> B3[Session 3...]

    B1 --> C1[Message 1]
    B1 --> C2[Message 2]
    B1 --> C3[Message 3...]

    C1 --> D1[Attachment 1]
    C1 --> D2[Attachment 2...]

    %% Adjusted purple for Agent with better contrast in light mode
    classDef agent fill:#c084fc,stroke:#000,stroke-width:2px,color:#000,font-weight:bold;
    classDef session fill:#3b82f6,stroke:#000,stroke-width:1px,color:#000,font-weight:bold;
    classDef message fill:#10b981,stroke:#000,stroke-width:1px,color:#000,font-weight:bold;
    classDef attachment fill:#f59e0b,stroke:#000,stroke-width:1px,color:#000,font-weight:bold;

    class A agent;
    class B1,B2,B3 session;
    class C1,C2,C3 message;
    class D1,D2 attachment;

This hierarchical structure forms the backbone of agent interactions:

Agent: A persistent AI entity with its own identity that evolves over time
Session: A distinct interaction context with that agent (like an email thread, not a stateless chat)
Message: Communication within a session
Attachment: Rich media and files that can be included with messages

Persistent Identity

Unlike typical chatbots where each session starts from scratch, agents in this runtime automatically maintain and build upon context across all sessions. Think of interacting with your agent more like an ongoing email thread with a coworker who remembers your previous conversations, rather than a stateless chat session that forgets everything when you close it.

Creating Agents, Sessions and Messages¶

Creating an Agent¶

Here's how you can create an agent via the API:

curl -X POST https://api.tiptreesystems.com/platform/api/v2/actx/agents \
  -H 'Content-Type: application/json' \
  -d '{
    "config": {
      "name": "MyAssistantAgent",
      "description": "A helpful assistant that remembers our conversations"
    },
    "info": {
      "user_id": "user_12345",
      "custom_metadata": "any additional information"
    }
  }'

The info can store metadata you need for your application. This is useful for integrating with your application's data model or storing custom information that helps track or organize your agents and sessions.

API Response:

When you create an agent, the API returns an Agent object with the following structure:

{
  "id": "agent_abc123",         // Unique identifier for the agent
  "config": {                   // The configuration you provided
    "name": "MyAssistantAgent",
    "description": "A helpful assistant that remembers our conversations"
  },
  "info": {                     // The metadata you provided
    "user_id": "user_12345",
    "custom_metadata": "any additional information"
  },
  "created_at": 1683042267.32   // Timestamp when the agent was created
}

The returned id is important as you'll need it to create sessions, retrieve the agent later, or perform other operations with this agent.

You can retrieve an existing agent using:

curl -X GET https://api.tiptreesystems.com/platform/api/v2/actx/agents/{agent_id}

Or update an agent's configuration with:

curl -X PATCH https://api.tiptreesystems.com/platform/api/v2/actx/agents/{agent_id} \
  -H 'Content-Type: application/json' \
  -d '{
    "config": {
      "name": "Updated Agent Name",
      "description": "New description for this agent"
    }
  }'

Creating a Session with an Agent¶

Here's how you can create a session with an agent via the API:

curl -X POST https://api.tiptreesystems.com/platform/api/v2/actx/agents/{agent_id}/agent-sessions \
  -H 'Content-Type: application/json' \
  -d '{
    "config": {
      "initial_prompt": "For this session, please respond in a concise way. I appreciate directness and brevity."
    },
    "info": {
      "session_purpose": "Problem-solving and decision support"
    }
  }'

About Initial Prompt

The initial_prompt field is primarily used to provide context to the agent for this session, rather than specific instructions. It helps set the stage for the interaction. Actual instructions are better sent as messages within the session.

API Response:

When you create a session, the API returns an AgentSession object with the following structure:

{
  "id": "session_xyz789",       // Unique identifier for the session
  "config": {                   // The configuration you provided
    "initial_prompt": "For this session, please respond in a concise way. I appreciate directness and brevity."
  },
  "info": {                     // The metadata you provided
    "session_purpose": "Problem-solving and decision support"
  },
  "agent_id": "agent_abc123",   // ID of the agent this session belongs to
  "created_at": 1683042300.45   // Timestamp when the session was created
}

The returned id is important as you'll need it to send messages, wake the session, or perform other operations within this session.

You can retrieve an existing session using:

curl -X GET https://api.tiptreesystems.com/platform/api/v2/actx/agent-sessions/{agent_session_id}

Or update a session's configuration with:

curl -X PATCH https://api.tiptreesystems.com/platform/api/v2/actx/agent-sessions/{agent_session_id} \
  -H 'Content-Type: application/json' \
  -d '{
    "config": {
      "initial_prompt": "Conciseness, brevity, and directness are appreciated. Please push back if think I am wrong about something."
    }
  }'

Once you've created a session, you can start sending messages to it:

curl -X POST https://api.tiptreesystems.com/platform/api/v2/actx/agent-sessions/{agent_session_id}/messages \
  -H 'Content-Type: application/json' \
  -d '{
    "payload": {
      "content": "Can you please explain ...",
    }
  }'

Waking a Session¶

After sending a message to the agent session, you need to wake it up to start processing the message. Here's how to wake a session:

curl -X POST https://api.tiptreesystems.com/platform/api/v2/actx/agent-sessions/{agent_session_id}/wake

API Response:

The API returns an AgentSessionWake object:

{
  "agent_session_id": "session_xyz789",  // ID of the session that was woken
  "woken_at": 1683042350.78,             // Timestamp when the session was woken
  "success": true                        // Whether the wake operation was successful
}

Session Locking

Only one instance of a session can be awake at a time. This locking mechanism ensures consistency in the agent's state and prevents race conditions. If you try to wake a session that's already awake, the API will enqueue the wake request and return immediately. The session will be woken up asynchronously in the background once the previous wake operation completes.

Once a session is awake, the message or messages you sent to it will be processed. Once done, the session will go back to sleep automatically.

Polling for Messages¶

Since agent sessions are asynchronous, you'll need to poll for new messages after sending a request. Here's how to retrieve messages from a session:

curl -X GET https://api.tiptreesystems.com/platform/api/v2/actx/agent-sessions/{agent_session_id}/messages \
  -H 'Content-Type: application/json'

You can use query parameters to control pagination, sorting and filtering:

curl -X GET "https://api.tiptreesystems.com/platform/api/v2/actx/agent-sessions/{agent_session_id}/messages?offset=0&limit=10&most_recent_first=true"

Filtering and Sorting

The available filter parameters include:

offset and limit: For pagination control
most_recent_first: For sorting by creation time (descending when true)
sender: Filter messages by who sent them (e.g., "user", "assistant")
created_after and created_before: Filter by timestamp (Unix timestamp in seconds)
read: Filter by read status (true/false). When set to false, only unread messages are returned.

This allows you to efficiently retrieve exactly the messages you need, such as only unread messages from the assistant, or messages created within a specific time window.

API Response:

The API returns an array of Message objects:

[
  {
    "id": "msg_123abc",                // Unique identifier for the message
    "payload": {                       // The content and metadata of the message
      "content": "Can you help me research electric vehicles?",
      "sender": "user",
      "attachments": []
    },
    "agent_session_id": "session_xyz789", // ID of the session this message belongs to
    "created_at": 1683042400.12,        // Timestamp when the message was created
    "read": false                       // Whether the message has been marked as read
  },
  {
    "id": "msg_456def",
    "payload": {
      "content": "I'd be happy to help you research electric vehicles. What specific aspects are you interested in learning about?",
      "sender": "assistant",
      "attachments": []
    },
    "agent_session_id": "session_xyz789",
    "created_at": 1683042430.45,
    "read": false
  }
]

You can also mark messages as read once you've processed them:

curl -X PATCH https://api.tiptreesystems.com/platform/api/v2/actx/agent-sessions/{agent_session_id}/messages/{message_id} \
  -H 'Content-Type: application/json' \
  -d '{
    "read": true
  }'

Marking Messages as Read

Marking messages as read will prevent them from being returned in future message polls if you set the read parameter to false.

If you prefer real-time updates instead of polling, you can stream events from an awake session:

curl -X GET https://api.tiptreesystems.com/platform/api/v2/actx/agent-sessions/{agent_session_id}/events \
  -H 'Accept: text/event-stream'

This will provide a stream of events including new messages as they occur, which can be more efficient than continuous polling depending on your use case.

Understanding Memory and Context¶

One of the most powerful features of the Agent Runtime Platform is how agents maintain memory across sessions, creating opportunities for a continuous, evolving relationship with users. This means:

Sessions are not siloed, unlike in existing chat applications. The agent seamlessly can use context between sessions, creating a continuous experience across all interactions.
In the one-user-one-agent model, the agent remembers over all conversations and builds a knowledge bank over time. This means your agent becomes increasingly personalized and effective as it learns your preferences, communication style, and needs.
For ephemeral interactions, you probably want to initialize a new agent for each interaction. This approach ensures no context is carried forward when privacy or separation of concerns is important.

graph LR
    User((Human User)) --- Agent

    Agent --- S1[Session 1]
    Agent --- S2[Session 2]
    Agent --- S3[Session 3]

    S1 --- M1[Context]
    S2 --- M2[Context]
    S3 --- M3[Context]

    M1 --> MB((Memory Bank))
    M2 --> MB
    M3 --> MB

    MB -.-> S1
    MB -.-> S2
    MB -.-> S3

    classDef user fill:#f9f,stroke:#333,stroke-width:2px;
    classDef agent fill:#c084fc,stroke:#000,stroke-width:2px;
    classDef session fill:#3b82f6,stroke:#000,stroke-width:1px;
    classDef memory fill:#10b981,stroke:#000,stroke-width:1px;
    classDef knowledge fill:#f59e0b,stroke:#000,stroke-width:1px;

    class User user;
    class Agent agent;
    class S1,S2,S3 session;
    class M1,M2,M3 memory;
    class MB knowledge;

Memory Bank

The memory bank is an indexed collection of all the memories of the agent. Under the hood, it's an Elasticsearch index used for both vector and hybrid search.

Think of your agent as a colleague who remembers your previous conversations from days, weeks, or even months ago. When you mention something related to a past discussion, the agent can naturally reference it without needing explicit reminders—even if that discussion happened in a completely different session.

How does it work?

Each agent is equipped with a sub-agent that continuously gathers and summarizes context. This approach is inspired by the concept of "Remembrance Agents", which are proactive systems that autonomously monitor user activities and provide relevant information without explicit requests.

The sub-agent works in the background to analyze message content and stored memories across all sessions. It then extracts key information and injects it into the main agent's context.

This allows the main agent to access relevant context at the right time, creating a seamless experience that feels like talking to someone who genuinely remembers you.

TASX: The Agent Service eXchange¶

While ACTX provides the runtime environment for agents, TASX equips them with capabilities to interact with the world. Think of ACTX as the smartphone and TASX as the app store—a rich ecosystem of tasks and tools that agents can discover and use.

What is a task?¶

Tasks are more than just tools in the traditional sense. A task can be:

Long-running: Tasks can run for days or weeks, unlike traditional function calls.
Interactive: Tasks can engage in back-and-forth communication with the agent.
Stateful: Tasks maintain their state between interactions, allowing them to be paused and resumed.
Complex: Tasks can represent multi-step workflows or entire business processes.

Tasks vs. Tools

Tools are a specific type of task—they're synchronous and blocking, meaning the agent waits for them to complete before continuing. Most tasks in TASX are asynchronous and non-blocking, allowing the agent to perform other activities while the task runs in the background.

Tasks as Agents

TASX is designed so that ACTX agents themselves can be tasks. This enables agent-to-agent collaboration, where one user's agent can interact with another user's agent to accomplish shared goals.

Task Organization¶

Tasks in TASX are organized in a three-level hierarchy:

Provider: The entity that hosts and executes the tasks. This could be a default provider hosted by Tiptree Systems, or a custom provider you deploy that integrates with your own systems.
Service: A logical grouping of related tasks, typically representing a domain or capability area.
Task: The specific capability or function that an agent can use.

graph TD
    subgraph Provider[Provider: tiptree-systems]
        subgraph WebBrowser[Service: web-browser]
            T1[Task: open-url]
            T2[Task: answer-from-url]
            T3[Task: get-relevant-urls]
            T3_[Task: ...]
        end

        subgraph DataAnalysis[Service: data-analysis]
            T4[Task: analyze-csv]
            T5[Task: generate-chart]
            T5_[Task: ...]
        end

        subgraph OtherService[Service: ...]
            T8[Task: ...]
        end
    end

    subgraph CustomProvider[Provider: your-company]
        subgraph CustomService[Service: internal-tools]
            T6[Task: query-database]
            T7[Task: update-crm]
            T7_[Task: ...]
        end

        subgraph CustomService2[Service: ...]
            T9[Task: ...]
        end
    end

    subgraph OtherProvider[Provider: ...]
        subgraph OtherService2[Service: ...]
            T10[Task: ...]
        end
    end

    %% Style definitions
    classDef provider fill:#c084fc,stroke:#000,stroke-width:2px,color:#000;
    classDef service fill:#3b82f6,stroke:#000,stroke-width:1px,color:#000;
    classDef task fill:#10b981,stroke:#000,stroke-width:1px,color:#000;

    class Provider,CustomProvider,OtherProvider provider;
    class WebBrowser,DataAnalysis,CustomService,OtherService,CustomService2,OtherService2 service;
    class T1,T2,T3,T3_,T4,T5,T5_,T6,T7,T7_,T8,T9,T10 task;

For example, tiptree-systems/web-browser/open-url refers to the "open-url" task in the "web-browser" service hosted by the "tiptree-systems" provider. This task can be used by an agent to open a URL and read its content as a markdown document. Other tasks in the same service include answer-from-url (to answer a question based on the content of a URL) or get-relevant-urls (to find other URLs in the web page that are relevant to the question).

This addressing scheme also allows for tasks that are hosted by third parties (and even ones that you host locally on your own infrastructure). They would have the URL:

tasxs://example.com/service-name/task-name

Hosting a Provider

Tiptree Systems hosts a default provider with common tools, but you can deploy your own providers with custom capabilities. Hosting a provider is as simple as hosting an API server that can execute the tasks, e.g. with FastAPI or any other framework.

For enterprise use cases, the API can be hosted on-premise or in a private cloud. Contact us to discuss your use case.

Discovering and Using Tasks¶

A key feature of TASX is its discoverability. Each provider implements a standardized search interface that allows ACTX agents to find the right task for their needs, enabling them to:

Discover relevant tasks dynamically: Rather than being limited to a predefined set of tools, agents can search for and use the most appropriate task based on the current context.
Adapt to new capabilities: As new tasks are added to providers, agents can immediately discover and use them without requiring updates to the agent itself.
Find specialized tools: When faced with domain-specific challenges, agents can search for specialized tasks that handle those particular use cases.

The Default Provider

The default provider for ACTX agents provides a growing set of essential tasks, including web search and browsing, data analysis, and more. This means you can start using ACTX agents right away while we continue to build out even more capabilities.

What makes this architecture particularly powerful is its ability to scale with the growing ecosystem of tasks. When new tasks are added to any provider, ACTX agents can immediately discover and utilize them without requiring any updates to the agent itself. But the real power comes from the agents' ability to intelligently combine and orchestrate these tasks.

For example, when an agent needs to research a topic, it might:

Use a search task (tiptree-systems/web-search/search) to find relevant URLs
Use a URL reader task (tiptree-systems/web-browser/read-url) to extract content from those pages
Use a summarization task (tiptree-systems/web-browser/answer-from-url) to distill the key information
Finally, present the findings to the user

This intelligent task composition happens dynamically based on the context and requirements. If one approach fails (e.g., if a website is inaccessible), the agent can automatically adapt and try alternative strategies using different combinations of available tasks.

Scalable Architecture

Unlike traditional approaches where all available tools are frontloaded into the agent's context window (which quickly becomes a bottleneck), ACTX's dynamic task discovery allows the system to scale gracefully as the number of available tasks grows. Agents can access an ever-expanding toolkit while maintaining efficient context management.

The default provider¶

The default provider is a special task provider operated by Tiptree Systems that is available by default to all ACTX agents. It includes a set of core tasks that are useful for a wide range of use cases:

Web Browsing: Search the web, read web pages, and answer questions based on the content of web pages. Tasks include:
- tiptree-systems/web-search/search: Search the web for relevant URLs
- tiptree-systems/web-browser/open-url: Open a URL and read its content as a markdown document
- tiptree-systems/web-browser/answer-from-url: Answer a question based on the content of a web page
Research: Find and analyze information from a variety of sources. Tasks include:
- tiptree-systems/research/web-research: Research a topic by searching the web and synthesizing the results
- tiptree-systems/research/search-patents: Search for patents by topic and other criteria
- tiptree-systems/research/search-publications: Search for academic papers by topic and other criteria
- tiptree-systems/research/thorough-web-research: Perform a thorough and deep web research by iteratively searching the web, reading web pages, and answering questions based on the content of web pages. This is an example of an asynchronous task that can run for minutes.
Prospecting: Find and analyze potential customers and leads.
Social Media: Search and analyze content on social media platforms.

What's next?¶

Get started with the platform or reach out to us at nasim@tiptreesystems.com if you have any questions.