autogen/notebook/agentchat_agentops.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "abb8a01d85d8b146",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Agent Tracking with AgentOps"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a447802c88c8a240",
   "metadata": {},
   "source": [
    "<img src=\"https://github.com/AgentOps-AI/agentops/blob/main/docs/images/external/logo/banner-badge.png?raw=true\"/>\n",
    "\n",
    "[AgentOps](https://agentops.ai/?=autogen) provides session replays, metrics, and monitoring for AI agents.\n",
    "\n",
    "At a high level, AgentOps gives you the ability to monitor LLM calls, costs, latency, agent failures, multi-agent interactions, tool usage, session-wide statistics, and more. For more info, check out the [AgentOps Repo](https://github.com/AgentOps-AI/agentops).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b354c068",
   "metadata": {},
   "source": [
    "### Overview Dashboard\n",
    "<img src=\"https://raw.githubusercontent.com/AgentOps-AI/agentops/main/docs/images/external/app_screenshots/overview.gif\"/>\n",
    "\n",
    "### Session Replays\n",
    "<img src=\"https://raw.githubusercontent.com/AgentOps-AI/agentops/main/docs/images/external/app_screenshots/drilldown.gif\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38182a5296dceb34",
   "metadata": {},
   "source": [
    "## Adding AgentOps to an existing Autogen service.\n",
    "To get started, you'll need to install the AgentOps package and set an API key.\n",
    "\n",
    "AgentOps automatically configures itself when it's initialized meaning your agent run data will be tracked and logged to your AgentOps account right away."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d9451f4",
   "metadata": {},
   "source": [
    "````{=mdx}\n",
    ":::info Requirements\n",
    "Some extra dependencies are needed for this notebook, which can be installed via pip:\n",
    "\n",
    "```bash\n",
    "pip install autogen-agentchat~=0.2 agentops\n",
    "```\n",
    "\n",
    "For more information, please refer to the [installation guide](/docs/installation/).\n",
    ":::\n",
    "````"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6be9e11620b0e8d6",
   "metadata": {},
   "source": [
    "### Set an API key\n",
    "\n",
    "By default, the AgentOps `init()` function will look for an environment variable named `AGENTOPS_API_KEY`. Alternatively, you can pass one in as an optional parameter.\n",
    "\n",
    "Create an account and obtain an API key at [AgentOps.ai](https://agentops.ai/settings/projects)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f31a28d20a13b377",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-31T22:48:27.679318Z",
     "start_time": "2024-05-31T22:48:26.192071Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "🖇 AgentOps: \u001b[34m\u001b[34mSession Replay: https://app.agentops.ai/drilldown?session_id=8bfaeed1-fd51-4c68-b3ec-276b1a3ce8a4\u001b[0m\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "UUID('8bfaeed1-fd51-4c68-b3ec-276b1a3ce8a4')"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import agentops\n",
    "\n",
    "from autogen import ConversableAgent, UserProxyAgent, config_list_from_json\n",
    "\n",
    "agentops.init(api_key=\"...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dd8f461ccd9cbef",
   "metadata": {},
   "source": [
    "Autogen will now start automatically tracking\n",
    "- LLM prompts and completions\n",
    "- Token usage and costs\n",
    "- Agent names and actions\n",
    "- Correspondence between agents\n",
    "- Tool usage\n",
    "- Errors"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "712315c520536eb8",
   "metadata": {},
   "source": [
    "# Simple Chat Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "66d68e66e9f4a677",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-31T22:48:32.813123Z",
     "start_time": "2024-05-31T22:48:27.677564Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33magent\u001b[0m (to user):\n",
      "\n",
      "How can I help you today?\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "\u001b[33muser\u001b[0m (to agent):\n",
      "\n",
      "2+2\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "\u001b[31m\n",
      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
      "\u001b[33magent\u001b[0m (to user):\n",
      "\n",
      "2 + 2 equals 4.\n",
      "\n",
      "--------------------------------------------------------------------------------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "🖇 AgentOps: This run's cost $0.000960\n",
      "🖇 AgentOps: \u001b[34m\u001b[34mSession Replay: https://app.agentops.ai/drilldown?session_id=8bfaeed1-fd51-4c68-b3ec-276b1a3ce8a4\u001b[0m\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "import agentops\n",
    "\n",
    "# When initializing AgentOps, you can pass in optional tags to help filter sessions\n",
    "agentops.init(tags=[\"simple-autogen-example\"])\n",
    "\n",
    "# Create the agent that uses the LLM.\n",
    "config_list = config_list_from_json(env_or_file=\"OAI_CONFIG_LIST\")\n",
    "assistant = ConversableAgent(\"agent\", llm_config={\"config_list\": config_list})\n",
    "\n",
    "# Create the agent that represents the user in the conversation.\n",
    "user_proxy = UserProxyAgent(\"user\", code_execution_config=False)\n",
    "\n",
    "# Let the assistant start the conversation.  It will end when the user types \"exit\".\n",
    "assistant.initiate_chat(user_proxy, message=\"How can I help you today?\")\n",
    "\n",
    "# Close your AgentOps session to indicate that it completed.\n",
    "agentops.end_session(\"Success\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2217ed0f930cfcaa",
   "metadata": {},
   "source": [
    "You can view data on this run at [app.agentops.ai](https://app.agentops.ai). \n",
    "\n",
    "The dashboard will display LLM events for each message sent by each agent, including those made by the human user."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbd689b0f5617013",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "![session replay](https://github.com/AgentOps-AI/agentops/blob/main/docs/images/external/app_screenshots/session-overview.png?raw=true)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd78f1a816276cb7",
   "metadata": {},
   "source": [
    "# Tool Example\n",
    "AgentOps also tracks when Autogen agents use tools. You can find more information on this example in [tool-use.ipynb](https://github.com/microsoft/autogen/blob/main/website/docs/tutorial/tool-use.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3498aa6176c799ff",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-31T22:48:35.808674Z",
     "start_time": "2024-05-31T22:48:32.813225Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "🖇 AgentOps: \u001b[34m\u001b[34mSession Replay: https://app.agentops.ai/drilldown?session_id=880c206b-751e-4c23-9313-8684537fc04d\u001b[0m\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mUser\u001b[0m (to Assistant):\n",
      "\n",
      "What is (1423 - 123) / 3 + (32 + 23) * 5?\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "\u001b[31m\n",
      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
      "\u001b[33mAssistant\u001b[0m (to User):\n",
      "\n",
      "\u001b[32m***** Suggested tool call (call_aINcGyo0Xkrh9g7buRuhyCz0): calculator *****\u001b[0m\n",
      "Arguments: \n",
      "{\n",
      "  \"a\": 1423,\n",
      "  \"b\": 123,\n",
      "  \"operator\": \"-\"\n",
      "}\n",
      "\u001b[32m***************************************************************************\u001b[0m\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "\u001b[35m\n",
      ">>>>>>>> EXECUTING FUNCTION calculator...\u001b[0m\n",
      "\u001b[33mUser\u001b[0m (to Assistant):\n",
      "\n",
      "\u001b[33mUser\u001b[0m (to Assistant):\n",
      "\n",
      "\u001b[32m***** Response from calling tool (call_aINcGyo0Xkrh9g7buRuhyCz0) *****\u001b[0m\n",
      "1300\n",
      "\u001b[32m**********************************************************************\u001b[0m\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "\u001b[31m\n",
      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
      "\u001b[33mAssistant\u001b[0m (to User):\n",
      "\n",
      "\u001b[32m***** Suggested tool call (call_prJGf8V0QVT7cbD91e0Fcxpb): calculator *****\u001b[0m\n",
      "Arguments: \n",
      "{\n",
      "  \"a\": 1300,\n",
      "  \"b\": 3,\n",
      "  \"operator\": \"/\"\n",
      "}\n",
      "\u001b[32m***************************************************************************\u001b[0m\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "\u001b[35m\n",
      ">>>>>>>> EXECUTING FUNCTION calculator...\u001b[0m\n",
      "\u001b[33mUser\u001b[0m (to Assistant):\n",
      "\n",
      "\u001b[33mUser\u001b[0m (to Assistant):\n",
      "\n",
      "\u001b[32m***** Response from calling tool (call_prJGf8V0QVT7cbD91e0Fcxpb) *****\u001b[0m\n",
      "433\n",
      "\u001b[32m**********************************************************************\u001b[0m\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "\u001b[31m\n",
      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/braelynboynton/Developer/agentops/autogen/autogen/agentchat/conversable_agent.py:2489: UserWarning: Function 'calculator' is being overridden.\n",
      "  warnings.warn(f\"Function '{tool_sig['function']['name']}' is being overridden.\", UserWarning)\n",
      "/Users/braelynboynton/Developer/agentops/autogen/autogen/agentchat/conversable_agent.py:2408: UserWarning: Function 'calculator' is being overridden.\n",
      "  warnings.warn(f\"Function '{name}' is being overridden.\", UserWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mAssistant\u001b[0m (to User):\n",
      "\n",
      "\u001b[32m***** Suggested tool call (call_CUIgHRsySLjayDKuUphI1TGm): calculator *****\u001b[0m\n",
      "Arguments: \n",
      "{\n",
      "  \"a\": 32,\n",
      "  \"b\": 23,\n",
      "  \"operator\": \"+\"\n",
      "}\n",
      "\u001b[32m***************************************************************************\u001b[0m\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "\u001b[35m\n",
      ">>>>>>>> EXECUTING FUNCTION calculator...\u001b[0m\n",
      "\u001b[33mUser\u001b[0m (to Assistant):\n",
      "\n",
      "\u001b[33mUser\u001b[0m (to Assistant):\n",
      "\n",
      "\u001b[32m***** Response from calling tool (call_CUIgHRsySLjayDKuUphI1TGm) *****\u001b[0m\n",
      "55\n",
      "\u001b[32m**********************************************************************\u001b[0m\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "\u001b[31m\n",
      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
      "\u001b[33mAssistant\u001b[0m (to User):\n",
      "\n",
      "\u001b[32m***** Suggested tool call (call_L7pGtBLUf9V0MPL90BASyesr): calculator *****\u001b[0m\n",
      "Arguments: \n",
      "{\n",
      "  \"a\": 55,\n",
      "  \"b\": 5,\n",
      "  \"operator\": \"*\"\n",
      "}\n",
      "\u001b[32m***************************************************************************\u001b[0m\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "\u001b[35m\n",
      ">>>>>>>> EXECUTING FUNCTION calculator...\u001b[0m\n",
      "\u001b[33mUser\u001b[0m (to Assistant):\n",
      "\n",
      "\u001b[33mUser\u001b[0m (to Assistant):\n",
      "\n",
      "\u001b[32m***** Response from calling tool (call_L7pGtBLUf9V0MPL90BASyesr) *****\u001b[0m\n",
      "275\n",
      "\u001b[32m**********************************************************************\u001b[0m\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "\u001b[31m\n",
      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
      "\u001b[33mAssistant\u001b[0m (to User):\n",
      "\n",
      "\u001b[32m***** Suggested tool call (call_Ygo6p4XfcxRjkYBflhG3UVv6): calculator *****\u001b[0m\n",
      "Arguments: \n",
      "{\n",
      "  \"a\": 433,\n",
      "  \"b\": 275,\n",
      "  \"operator\": \"+\"\n",
      "}\n",
      "\u001b[32m***************************************************************************\u001b[0m\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "\u001b[35m\n",
      ">>>>>>>> EXECUTING FUNCTION calculator...\u001b[0m\n",
      "\u001b[33mUser\u001b[0m (to Assistant):\n",
      "\n",
      "\u001b[33mUser\u001b[0m (to Assistant):\n",
      "\n",
      "\u001b[32m***** Response from calling tool (call_Ygo6p4XfcxRjkYBflhG3UVv6) *****\u001b[0m\n",
      "708\n",
      "\u001b[32m**********************************************************************\u001b[0m\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "\u001b[31m\n",
      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
      "\u001b[33mAssistant\u001b[0m (to User):\n",
      "\n",
      "The result of the calculation is 708.\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "\u001b[33mUser\u001b[0m (to Assistant):\n",
      "\n",
      "\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "\u001b[31m\n",
      ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
      "\u001b[33mAssistant\u001b[0m (to User):\n",
      "\n",
      "TERMINATE\n",
      "\n",
      "--------------------------------------------------------------------------------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "🖇 AgentOps: This run's cost $0.001800\n",
      "🖇 AgentOps: \u001b[34m\u001b[34mSession Replay: https://app.agentops.ai/drilldown?session_id=880c206b-751e-4c23-9313-8684537fc04d\u001b[0m\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "from typing import Annotated, Literal\n",
    "\n",
    "from autogen import ConversableAgent, config_list_from_json, register_function\n",
    "\n",
    "agentops.start_session(tags=[\"autogen-tool-example\"])\n",
    "\n",
    "Operator = Literal[\"+\", \"-\", \"*\", \"/\"]\n",
    "\n",
    "\n",
    "def calculator(a: int, b: int, operator: Annotated[Operator, \"operator\"]) -> int:\n",
    "    if operator == \"+\":\n",
    "        return a + b\n",
    "    elif operator == \"-\":\n",
    "        return a - b\n",
    "    elif operator == \"*\":\n",
    "        return a * b\n",
    "    elif operator == \"/\":\n",
    "        return int(a / b)\n",
    "    else:\n",
    "        raise ValueError(\"Invalid operator\")\n",
    "\n",
    "\n",
    "config_list = config_list_from_json(env_or_file=\"OAI_CONFIG_LIST\")\n",
    "\n",
    "# Create the agent that uses the LLM.\n",
    "assistant = ConversableAgent(\n",
    "    name=\"Assistant\",\n",
    "    system_message=\"You are a helpful AI assistant. \"\n",
    "    \"You can help with simple calculations. \"\n",
    "    \"Return 'TERMINATE' when the task is done.\",\n",
    "    llm_config={\"config_list\": config_list},\n",
    ")\n",
    "\n",
    "# The user proxy agent is used for interacting with the assistant agent\n",
    "# and executes tool calls.\n",
    "user_proxy = ConversableAgent(\n",
    "    name=\"User\",\n",
    "    llm_config=False,\n",
    "    is_termination_msg=lambda msg: msg.get(\"content\") is not None and \"TERMINATE\" in msg[\"content\"],\n",
    "    human_input_mode=\"NEVER\",\n",
    ")\n",
    "\n",
    "assistant.register_for_llm(name=\"calculator\", description=\"A simple calculator\")(calculator)\n",
    "user_proxy.register_for_execution(name=\"calculator\")(calculator)\n",
    "\n",
    "# Register the calculator function to the two agents.\n",
    "register_function(\n",
    "    calculator,\n",
    "    caller=assistant,  # The assistant agent can suggest calls to the calculator.\n",
    "    executor=user_proxy,  # The user proxy agent can execute the calculator calls.\n",
    "    name=\"calculator\",  # By default, the function name is used as the tool name.\n",
    "    description=\"A simple calculator\",  # A description of the tool.\n",
    ")\n",
    "\n",
    "# Let the assistant start the conversation.  It will end when the user types \"exit\".\n",
    "user_proxy.initiate_chat(assistant, message=\"What is (1423 - 123) / 3 + (32 + 23) * 5?\")\n",
    "\n",
    "agentops.end_session(\"Success\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b4edf8e70d17267",
   "metadata": {},
   "source": [
    "You can see your run in action at [app.agentops.ai](https://app.agentops.ai). In this example, the AgentOps dashboard will show:\n",
    "- Agents talking to each other\n",
    "- Each use of the `calculator` tool\n",
    "- Each call to OpenAI for LLM use"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a922a52ab5fce31",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "![Session Drilldown](https://github.com/AgentOps-AI/agentops/blob/main/docs/images/external/app_screenshots/session-replay.png?raw=true)"
   ]
  }
 ],
 "metadata": {
  "front_matter": {
   "description": "Use AgentOps to simplify the development process and monitor your agents in production.",
   "tags": [
    "monitoring",
    "debugging"
   ]
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}