I had an idea of what I wanted to accomplish (why I wanted to build Mirgenta), and how realistic it was to achieve (via MIR). My next step was deciding on what direction to take.
I spent one week familiarizing myself with the Music Information Retrieval landscape (MIREX, ISMIR, ICASSP) -- then I realized that the field was always evolving. In order for an Agent to have specialized knowledge over an evolving domain, I decided to build a RAG pipeline.
I decided on using pgvector as my vector database because I wanted to keep the infrastructure as simple as possible. I was going to need a relational database anyways for the storage layer, and for a prototype project, pgvector is powerful enough. I didn't want to manage a separate vector db like Pinecone or Weaviate or ChromaDB.
My RAG pipeline ingested, chunked, and embedded relevant MIR papers from arXiv.org. To do this, I utilized HuggingFace Hub API for discovery of papers. I settled on a clean 1000 papers to start off my corpus with. I applied for SemanticScholar's API access to get their Citation Graph Discovery feature, but they didn't provide me access. I also found 21 open source libraries which could perform MIR tasks and added them to the RAG corpus.
For the embedding model, I settled on the Qwen3-Embedding series (Qwen3-Embedding-8B, Qwen3-Embedding-4B, Qwen3-Embedding-0.6B). For the 8B and 4B, I performed matryoshka truncation in order to be compatible with pgvector's HNSW indexing. The initial goal was to compare the performance-cost tradeoffs of these embedding model sizes later. For now, I am only utilizing the 0.6B model because I don't have the compute to run the others.
One question to ask about any agent system is how a human would interact with it. Would the agent be fully autonomous? Would it be semi-autonomous? Given my initial wish for an AI agent that could be your listening companion, I knew I needed an interface for a human to interact with the agent, which is why I settled on the Chatgpt/Claude style app. To me, this made the most sense.
And thus I began designing the initial database schema.
I knew I needed a users table, as I wanted users to be able to persist their conversations.
For the conversations, I named them sessions, and each session consisted of messages, which I called turns.
Okay this is cool and all, but Chatgpt/Claude is not an AI Agent. That's a simple LLM application. So where does the Agentic part come in?
To me, one major use case of an AI Agent is for it to perform tasks over a long horizon. A conversational-style chatbot necessitates a relatively quick feedback loop. This is not the case for all AI Agents.
So I added steps to agent turns. Each step could either be a reasoning step, a tool call, or a deterministic step (if-else statement) in conjunction with the current context. I am utilizing a DAG as the underlying structure.
Why am I using a DAG when Agents are actually running in loops? The primary reason is that I consider the state as part of the uniqueness of a node. A cyclical graph would define the underlying structure of an Agent loop, which is what LangChain and LangGraph does. But the system I've built focuses more on dynamic runtime generation that aims to record what steps were taken.