Overview

Figure: Architecture of R6D9-Virtual Multi-Agentic System for Computer Use

1.1 Mission

Robotic agents are AI-driven systems that sense, decide, and act to complete tasks in both virtual and real environments. By automating repetitive and complex operations, they enhance human productivity, freeing people to focus on higher-value work.

R6D9 aims to build a unified framework for robotic agent development, simplifying their creation, deployment, and continuous improvement through training data. By standardizing core components—policy training, environment perception, and data-driven adaptation—we enable agents to evolve, learn efficiently, and seamlessly integrate into diverse applications, from automating computer use to executing real-world tasks.

1.2 Robotic Agent in Virtual: Computer Use

Current computer operations still rely heavily on manual interaction. While automation tools such as RPA and macro recording exist, they often have high learning costs, limited generalization, and low adaptability.

This project aims to build a universal Computer Control Agent, allowing users to complete complex computer operations simply by describing tasks in natural language. This will lower the barrier to automation and improve work efficiency.The project will be developed in two phases at this time point:

• Browser Agent: Automates web interactions, including form filling, data extraction, auto-login, and data analysis.

• Computer Agent: Expands to desktop and mobile applications, enabling file management, software operation, and system task automation.

Ultimately, anyone, human and AI agent, will be able to control their computer with natural language, achieving efficient task automation without requiring programming knowledge.

1.2 Challenges

Despite advancements in AI automation, we still face the following key challenges:

1. Task Understanding & Decomposition

   1. How can AI fully comprehend user input in natural language and break it down into executable steps?

   2. Does the task require coordination across different environments (browser/desktop/mobile)?

2. Reliability of Execution

   1. How can we ensure automation accuracy and minimize errors?

   2. How can AI maintain stability despite UI changes, website updates, or software upgrades?

3. Security & Permission Management

   1. How can we prevent AI from triggering unintended or malicious actions?

   2. How can we protect user privacy and prevent sensitive data leaks?

4. Environmental Adaptability 🌍

   1. How can AI adapt to different software, web architectures, and operating systems?

   2. How can it reliably simulate human actions in the absence of APIs?

This project will integrate AI task planning, multimodal perception, and intelligent execution optimization to gradually overcome these challenges and build an efficient, stable, and secure Computer Control Agent.