mirror of https://github.com/microsoft/autogen.git
539 lines
19 KiB
Plaintext
539 lines
19 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "abb8a01d85d8b146",
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"source": [
|
|
"# Agent Tracking with AgentOps"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a447802c88c8a240",
|
|
"metadata": {},
|
|
"source": [
|
|
"<img src=\"https://github.com/AgentOps-AI/agentops/blob/main/docs/images/external/logo/banner-badge.png?raw=true\"/>\n",
|
|
"\n",
|
|
"[AgentOps](https://agentops.ai/?=autogen) provides session replays, metrics, and monitoring for AI agents.\n",
|
|
"\n",
|
|
"At a high level, AgentOps gives you the ability to monitor LLM calls, costs, latency, agent failures, multi-agent interactions, tool usage, session-wide statistics, and more. For more info, check out the [AgentOps Repo](https://github.com/AgentOps-AI/agentops).\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b354c068",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Overview Dashboard\n",
|
|
"<img src=\"https://raw.githubusercontent.com/AgentOps-AI/agentops/main/docs/images/external/app_screenshots/overview.gif\"/>\n",
|
|
"\n",
|
|
"### Session Replays\n",
|
|
"<img src=\"https://raw.githubusercontent.com/AgentOps-AI/agentops/main/docs/images/external/app_screenshots/drilldown.gif\"/>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "38182a5296dceb34",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Adding AgentOps to an existing Autogen service.\n",
|
|
"To get started, you'll need to install the AgentOps package and set an API key.\n",
|
|
"\n",
|
|
"AgentOps automatically configures itself when it's initialized meaning your agent run data will be tracked and logged to your AgentOps account right away."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8d9451f4",
|
|
"metadata": {},
|
|
"source": [
|
|
"````{=mdx}\n",
|
|
":::info Requirements\n",
|
|
"Some extra dependencies are needed for this notebook, which can be installed via pip:\n",
|
|
"\n",
|
|
"```bash\n",
|
|
"pip install autogen-agentchat~=0.2 agentops\n",
|
|
"```\n",
|
|
"\n",
|
|
"For more information, please refer to the [installation guide](/docs/installation/).\n",
|
|
":::\n",
|
|
"````"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6be9e11620b0e8d6",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Set an API key\n",
|
|
"\n",
|
|
"By default, the AgentOps `init()` function will look for an environment variable named `AGENTOPS_API_KEY`. Alternatively, you can pass one in as an optional parameter.\n",
|
|
"\n",
|
|
"Create an account and obtain an API key at [AgentOps.ai](https://agentops.ai/settings/projects)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "f31a28d20a13b377",
|
|
"metadata": {
|
|
"ExecuteTime": {
|
|
"end_time": "2024-05-31T22:48:27.679318Z",
|
|
"start_time": "2024-05-31T22:48:26.192071Z"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"🖇 AgentOps: \u001b[34m\u001b[34mSession Replay: https://app.agentops.ai/drilldown?session_id=8bfaeed1-fd51-4c68-b3ec-276b1a3ce8a4\u001b[0m\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"UUID('8bfaeed1-fd51-4c68-b3ec-276b1a3ce8a4')"
|
|
]
|
|
},
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import agentops\n",
|
|
"\n",
|
|
"from autogen import ConversableAgent, UserProxyAgent, config_list_from_json\n",
|
|
"\n",
|
|
"agentops.init(api_key=\"...\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4dd8f461ccd9cbef",
|
|
"metadata": {},
|
|
"source": [
|
|
"Autogen will now start automatically tracking\n",
|
|
"- LLM prompts and completions\n",
|
|
"- Token usage and costs\n",
|
|
"- Agent names and actions\n",
|
|
"- Correspondence between agents\n",
|
|
"- Tool usage\n",
|
|
"- Errors"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "712315c520536eb8",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Simple Chat Example"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "66d68e66e9f4a677",
|
|
"metadata": {
|
|
"ExecuteTime": {
|
|
"end_time": "2024-05-31T22:48:32.813123Z",
|
|
"start_time": "2024-05-31T22:48:27.677564Z"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[33magent\u001b[0m (to user):\n",
|
|
"\n",
|
|
"How can I help you today?\n",
|
|
"\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"\u001b[33muser\u001b[0m (to agent):\n",
|
|
"\n",
|
|
"2+2\n",
|
|
"\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"\u001b[31m\n",
|
|
">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
|
|
"\u001b[33magent\u001b[0m (to user):\n",
|
|
"\n",
|
|
"2 + 2 equals 4.\n",
|
|
"\n",
|
|
"--------------------------------------------------------------------------------\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"🖇 AgentOps: This run's cost $0.000960\n",
|
|
"🖇 AgentOps: \u001b[34m\u001b[34mSession Replay: https://app.agentops.ai/drilldown?session_id=8bfaeed1-fd51-4c68-b3ec-276b1a3ce8a4\u001b[0m\u001b[0m\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"import agentops\n",
|
|
"\n",
|
|
"# When initializing AgentOps, you can pass in optional tags to help filter sessions\n",
|
|
"agentops.init(tags=[\"simple-autogen-example\"])\n",
|
|
"\n",
|
|
"# Create the agent that uses the LLM.\n",
|
|
"config_list = config_list_from_json(env_or_file=\"OAI_CONFIG_LIST\")\n",
|
|
"assistant = ConversableAgent(\"agent\", llm_config={\"config_list\": config_list})\n",
|
|
"\n",
|
|
"# Create the agent that represents the user in the conversation.\n",
|
|
"user_proxy = UserProxyAgent(\"user\", code_execution_config=False)\n",
|
|
"\n",
|
|
"# Let the assistant start the conversation. It will end when the user types \"exit\".\n",
|
|
"assistant.initiate_chat(user_proxy, message=\"How can I help you today?\")\n",
|
|
"\n",
|
|
"# Close your AgentOps session to indicate that it completed.\n",
|
|
"agentops.end_session(\"Success\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2217ed0f930cfcaa",
|
|
"metadata": {},
|
|
"source": [
|
|
"You can view data on this run at [app.agentops.ai](https://app.agentops.ai). \n",
|
|
"\n",
|
|
"The dashboard will display LLM events for each message sent by each agent, including those made by the human user."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cbd689b0f5617013",
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"source": [
|
|
""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "fd78f1a816276cb7",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Tool Example\n",
|
|
"AgentOps also tracks when Autogen agents use tools. You can find more information on this example in [tool-use.ipynb](https://github.com/microsoft/autogen/blob/main/website/docs/tutorial/tool-use.ipynb)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "3498aa6176c799ff",
|
|
"metadata": {
|
|
"ExecuteTime": {
|
|
"end_time": "2024-05-31T22:48:35.808674Z",
|
|
"start_time": "2024-05-31T22:48:32.813225Z"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"🖇 AgentOps: \u001b[34m\u001b[34mSession Replay: https://app.agentops.ai/drilldown?session_id=880c206b-751e-4c23-9313-8684537fc04d\u001b[0m\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[33mUser\u001b[0m (to Assistant):\n",
|
|
"\n",
|
|
"What is (1423 - 123) / 3 + (32 + 23) * 5?\n",
|
|
"\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"\u001b[31m\n",
|
|
">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
|
|
"\u001b[33mAssistant\u001b[0m (to User):\n",
|
|
"\n",
|
|
"\u001b[32m***** Suggested tool call (call_aINcGyo0Xkrh9g7buRuhyCz0): calculator *****\u001b[0m\n",
|
|
"Arguments: \n",
|
|
"{\n",
|
|
" \"a\": 1423,\n",
|
|
" \"b\": 123,\n",
|
|
" \"operator\": \"-\"\n",
|
|
"}\n",
|
|
"\u001b[32m***************************************************************************\u001b[0m\n",
|
|
"\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"\u001b[35m\n",
|
|
">>>>>>>> EXECUTING FUNCTION calculator...\u001b[0m\n",
|
|
"\u001b[33mUser\u001b[0m (to Assistant):\n",
|
|
"\n",
|
|
"\u001b[33mUser\u001b[0m (to Assistant):\n",
|
|
"\n",
|
|
"\u001b[32m***** Response from calling tool (call_aINcGyo0Xkrh9g7buRuhyCz0) *****\u001b[0m\n",
|
|
"1300\n",
|
|
"\u001b[32m**********************************************************************\u001b[0m\n",
|
|
"\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"\u001b[31m\n",
|
|
">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
|
|
"\u001b[33mAssistant\u001b[0m (to User):\n",
|
|
"\n",
|
|
"\u001b[32m***** Suggested tool call (call_prJGf8V0QVT7cbD91e0Fcxpb): calculator *****\u001b[0m\n",
|
|
"Arguments: \n",
|
|
"{\n",
|
|
" \"a\": 1300,\n",
|
|
" \"b\": 3,\n",
|
|
" \"operator\": \"/\"\n",
|
|
"}\n",
|
|
"\u001b[32m***************************************************************************\u001b[0m\n",
|
|
"\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"\u001b[35m\n",
|
|
">>>>>>>> EXECUTING FUNCTION calculator...\u001b[0m\n",
|
|
"\u001b[33mUser\u001b[0m (to Assistant):\n",
|
|
"\n",
|
|
"\u001b[33mUser\u001b[0m (to Assistant):\n",
|
|
"\n",
|
|
"\u001b[32m***** Response from calling tool (call_prJGf8V0QVT7cbD91e0Fcxpb) *****\u001b[0m\n",
|
|
"433\n",
|
|
"\u001b[32m**********************************************************************\u001b[0m\n",
|
|
"\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"\u001b[31m\n",
|
|
">>>>>>>> USING AUTO REPLY...\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"/Users/braelynboynton/Developer/agentops/autogen/autogen/agentchat/conversable_agent.py:2489: UserWarning: Function 'calculator' is being overridden.\n",
|
|
" warnings.warn(f\"Function '{tool_sig['function']['name']}' is being overridden.\", UserWarning)\n",
|
|
"/Users/braelynboynton/Developer/agentops/autogen/autogen/agentchat/conversable_agent.py:2408: UserWarning: Function 'calculator' is being overridden.\n",
|
|
" warnings.warn(f\"Function '{name}' is being overridden.\", UserWarning)\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[33mAssistant\u001b[0m (to User):\n",
|
|
"\n",
|
|
"\u001b[32m***** Suggested tool call (call_CUIgHRsySLjayDKuUphI1TGm): calculator *****\u001b[0m\n",
|
|
"Arguments: \n",
|
|
"{\n",
|
|
" \"a\": 32,\n",
|
|
" \"b\": 23,\n",
|
|
" \"operator\": \"+\"\n",
|
|
"}\n",
|
|
"\u001b[32m***************************************************************************\u001b[0m\n",
|
|
"\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"\u001b[35m\n",
|
|
">>>>>>>> EXECUTING FUNCTION calculator...\u001b[0m\n",
|
|
"\u001b[33mUser\u001b[0m (to Assistant):\n",
|
|
"\n",
|
|
"\u001b[33mUser\u001b[0m (to Assistant):\n",
|
|
"\n",
|
|
"\u001b[32m***** Response from calling tool (call_CUIgHRsySLjayDKuUphI1TGm) *****\u001b[0m\n",
|
|
"55\n",
|
|
"\u001b[32m**********************************************************************\u001b[0m\n",
|
|
"\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"\u001b[31m\n",
|
|
">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
|
|
"\u001b[33mAssistant\u001b[0m (to User):\n",
|
|
"\n",
|
|
"\u001b[32m***** Suggested tool call (call_L7pGtBLUf9V0MPL90BASyesr): calculator *****\u001b[0m\n",
|
|
"Arguments: \n",
|
|
"{\n",
|
|
" \"a\": 55,\n",
|
|
" \"b\": 5,\n",
|
|
" \"operator\": \"*\"\n",
|
|
"}\n",
|
|
"\u001b[32m***************************************************************************\u001b[0m\n",
|
|
"\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"\u001b[35m\n",
|
|
">>>>>>>> EXECUTING FUNCTION calculator...\u001b[0m\n",
|
|
"\u001b[33mUser\u001b[0m (to Assistant):\n",
|
|
"\n",
|
|
"\u001b[33mUser\u001b[0m (to Assistant):\n",
|
|
"\n",
|
|
"\u001b[32m***** Response from calling tool (call_L7pGtBLUf9V0MPL90BASyesr) *****\u001b[0m\n",
|
|
"275\n",
|
|
"\u001b[32m**********************************************************************\u001b[0m\n",
|
|
"\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"\u001b[31m\n",
|
|
">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
|
|
"\u001b[33mAssistant\u001b[0m (to User):\n",
|
|
"\n",
|
|
"\u001b[32m***** Suggested tool call (call_Ygo6p4XfcxRjkYBflhG3UVv6): calculator *****\u001b[0m\n",
|
|
"Arguments: \n",
|
|
"{\n",
|
|
" \"a\": 433,\n",
|
|
" \"b\": 275,\n",
|
|
" \"operator\": \"+\"\n",
|
|
"}\n",
|
|
"\u001b[32m***************************************************************************\u001b[0m\n",
|
|
"\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"\u001b[35m\n",
|
|
">>>>>>>> EXECUTING FUNCTION calculator...\u001b[0m\n",
|
|
"\u001b[33mUser\u001b[0m (to Assistant):\n",
|
|
"\n",
|
|
"\u001b[33mUser\u001b[0m (to Assistant):\n",
|
|
"\n",
|
|
"\u001b[32m***** Response from calling tool (call_Ygo6p4XfcxRjkYBflhG3UVv6) *****\u001b[0m\n",
|
|
"708\n",
|
|
"\u001b[32m**********************************************************************\u001b[0m\n",
|
|
"\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"\u001b[31m\n",
|
|
">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
|
|
"\u001b[33mAssistant\u001b[0m (to User):\n",
|
|
"\n",
|
|
"The result of the calculation is 708.\n",
|
|
"\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"\u001b[33mUser\u001b[0m (to Assistant):\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"\u001b[31m\n",
|
|
">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
|
|
"\u001b[33mAssistant\u001b[0m (to User):\n",
|
|
"\n",
|
|
"TERMINATE\n",
|
|
"\n",
|
|
"--------------------------------------------------------------------------------\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"🖇 AgentOps: This run's cost $0.001800\n",
|
|
"🖇 AgentOps: \u001b[34m\u001b[34mSession Replay: https://app.agentops.ai/drilldown?session_id=880c206b-751e-4c23-9313-8684537fc04d\u001b[0m\u001b[0m\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from typing import Annotated, Literal\n",
|
|
"\n",
|
|
"from autogen import ConversableAgent, config_list_from_json, register_function\n",
|
|
"\n",
|
|
"agentops.start_session(tags=[\"autogen-tool-example\"])\n",
|
|
"\n",
|
|
"Operator = Literal[\"+\", \"-\", \"*\", \"/\"]\n",
|
|
"\n",
|
|
"\n",
|
|
"def calculator(a: int, b: int, operator: Annotated[Operator, \"operator\"]) -> int:\n",
|
|
" if operator == \"+\":\n",
|
|
" return a + b\n",
|
|
" elif operator == \"-\":\n",
|
|
" return a - b\n",
|
|
" elif operator == \"*\":\n",
|
|
" return a * b\n",
|
|
" elif operator == \"/\":\n",
|
|
" return int(a / b)\n",
|
|
" else:\n",
|
|
" raise ValueError(\"Invalid operator\")\n",
|
|
"\n",
|
|
"\n",
|
|
"config_list = config_list_from_json(env_or_file=\"OAI_CONFIG_LIST\")\n",
|
|
"\n",
|
|
"# Create the agent that uses the LLM.\n",
|
|
"assistant = ConversableAgent(\n",
|
|
" name=\"Assistant\",\n",
|
|
" system_message=\"You are a helpful AI assistant. \"\n",
|
|
" \"You can help with simple calculations. \"\n",
|
|
" \"Return 'TERMINATE' when the task is done.\",\n",
|
|
" llm_config={\"config_list\": config_list},\n",
|
|
")\n",
|
|
"\n",
|
|
"# The user proxy agent is used for interacting with the assistant agent\n",
|
|
"# and executes tool calls.\n",
|
|
"user_proxy = ConversableAgent(\n",
|
|
" name=\"User\",\n",
|
|
" llm_config=False,\n",
|
|
" is_termination_msg=lambda msg: msg.get(\"content\") is not None and \"TERMINATE\" in msg[\"content\"],\n",
|
|
" human_input_mode=\"NEVER\",\n",
|
|
")\n",
|
|
"\n",
|
|
"assistant.register_for_llm(name=\"calculator\", description=\"A simple calculator\")(calculator)\n",
|
|
"user_proxy.register_for_execution(name=\"calculator\")(calculator)\n",
|
|
"\n",
|
|
"# Register the calculator function to the two agents.\n",
|
|
"register_function(\n",
|
|
" calculator,\n",
|
|
" caller=assistant, # The assistant agent can suggest calls to the calculator.\n",
|
|
" executor=user_proxy, # The user proxy agent can execute the calculator calls.\n",
|
|
" name=\"calculator\", # By default, the function name is used as the tool name.\n",
|
|
" description=\"A simple calculator\", # A description of the tool.\n",
|
|
")\n",
|
|
"\n",
|
|
"# Let the assistant start the conversation. It will end when the user types \"exit\".\n",
|
|
"user_proxy.initiate_chat(assistant, message=\"What is (1423 - 123) / 3 + (32 + 23) * 5?\")\n",
|
|
"\n",
|
|
"agentops.end_session(\"Success\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2b4edf8e70d17267",
|
|
"metadata": {},
|
|
"source": [
|
|
"You can see your run in action at [app.agentops.ai](https://app.agentops.ai). In this example, the AgentOps dashboard will show:\n",
|
|
"- Agents talking to each other\n",
|
|
"- Each use of the `calculator` tool\n",
|
|
"- Each call to OpenAI for LLM use"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a922a52ab5fce31",
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"source": [
|
|
""
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"front_matter": {
|
|
"description": "Use AgentOps to simplify the development process and monitor your agents in production.",
|
|
"tags": [
|
|
"monitoring",
|
|
"debugging"
|
|
]
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.11.5"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|