Weave - A microkernel and layered architecture plugin/service development aggregation platform designed to provide high-performance, highly scalable, secure, and reliable plugin/service development

Language Options: 中文 | English

📋 Project Introduction

Weave from a simple thread to a complex tapestry, weaving is the creative process from simplicity to complexity. Developers can use Weave to develop various plugins or services, and through connection and integration, plugins and services can be seamlessly integrated and deeply fused, thereby building efficient and stable application systems. Weave, crafting infinite possibilities.

A high-performance, high-efficiency, pluggable, and easily extensible tool/service development platform based on Golang. It adopts a microkernel + layered architecture design, allowing developers to efficiently develop and easily integrate and manage various tools/services while maintaining system high performance and scalability.

Main application scenarios include:

Tool development and integration
Service development and aggregation
Data/service flow middleware platform
API gateway and service orchestration
Efficient development and prototyping platform

🏗️ Overall Architecture

Weave adopts a microkernel + layered architecture design pattern, fully leveraging the advantages of both architectures to ensure system availability and achieve high flexibility, scalability, and good performance.

Fusion of Microkernel and Layered Architecture

Weave integrates the design philosophy of layered architecture on the basis of the microkernel architecture, forming a complete, efficient, and flexible architectural system:

Microkernel Architecture (Plugin System): Provides plugin management, lifecycle control, and inter-plugin communication mechanisms.
Layered Architecture (Core System): Separates core functions by concerns, forming a clear hierarchical structure.

Microkernel Architecture Components

Core Kernel: Provides basic runtime environment, plugin management, configuration management, logging services, security mechanisms, and other basic functions.
Plugin System: The plugin manager is responsible for plugin registration, lifecycle management, dependency resolution, and conflict detection.
Extension Plugins: Integrated into the core system through plugin interfaces to implement various business functions.

Layered Architecture Components

Interface Layer: Handles HTTP requests, including route management and controllers.
Business Layer: Contains core business logic and the plugin system.
Data Layer: Responsible for data storage and access.
Infrastructure Layer: Provides services such as logging, configuration, and security.

Architectural Features

Loose Coupling Design: The core system and plugins communicate through well-defined interfaces, reducing inter-module dependencies.

Hot-Plug Capability: Plugins can be dynamically loaded and unloaded at runtime without restarting the system.

Functional Isolation: Each plugin independently encapsulates functionality, having its own namespace and route prefix.

Dependency and Conflict Management: Built-in dependency resolution and conflict detection mechanisms ensure harmonious coexistence among plugins.

Unified Interface: All plugins implement the same Plugin interface, standardizing the development process.

Extensibility: System functions can be extended on demand without modifying the kernel code.

Clear Hierarchy: The core system uses a layered design, with reasonable code organization, making it easy to maintain and extend.

High Performance: The layered design optimizes the request processing flow, improving system response speed.

The core of the system is an efficient and stable plugin mechanism and service aggregation, allowing functional modules to be independently developed and deployed as plugins/services, while interacting through unified interfaces. The overall architectural design emphasizes modularity, scalability, and high performance.

🌟 Project Features

🏗️ Microkernel + Layered Architecture

Stable Core & Clear Hierarchy: The core system remains minimal, and the layered design makes code organization reasonable, easy to maintain, and extend.
Flexible Functional Extension: Extend system functions on demand through the plugin mechanism without modifying the kernel code.
Low Coupling & High Cohesion: Loose coupling between system components facilitates maintenance and upgrades.
Hot-Plug Capability: Plugins can be dynamically loaded and unloaded at runtime without restarting the system.
Functional Isolation & Unified Management: Each plugin independently encapsulates functionality, has its own namespace and route prefix, while core services are uniformly managed through the layered architecture.
Dependency and Conflict Management: Built-in dependency resolution and conflict detection mechanisms ensure harmonious coexistence among plugins.
Unified Interface: All plugins implement the same Plugin interface, standardizing the development process.
High Performance: The layered design optimizes the request processing flow, improving system response speed.

🚀 High Performance/Efficiency

Built based on the Gin framework, offering fast request processing and strong concurrency capabilities.
Optimized database connection pool supporting high concurrent access.
Modular architecture design with clear code structure, easy to maintain and extend.
Supports environment variable overrides for easy configuration across different environments.
Efficient route management supporting dynamic routing and parameter binding.
The layered architecture optimizes the request processing flow, improving system response speed.

🔌 Pluggable & Easy to Extend

Unified plugin interface design supporting hot-plugging.
Plugin manager uniformly registers, manages, and executes plugins.
Plugins can independently register routes and have independent namespaces.
Plugin dependency and conflict detection mechanisms.
Scaffolding tool for conveniently generating plugin framework code.
Example plugins demonstrating the complete plugin development process.

🧠 Deep Service Integration

Seamlessly integrated with the microkernel architecture, highly extensible, and can be used as service aggregation/plugins.
For example, integrating services like LLM, RAG, etc., providing intelligent Q&A and document retrieval functions.
Efficient text retrieval based on RedisSearch.
Supports embedding, indexing, custom embedding models, and retrieval parameters for various document formats.
Provides flexible retrieval APIs for easy plugin integration.

🔒 Secure & Reliable

JWT-based authentication and authorization system.
Comprehensive CSRF protection mechanism.
Rate limiting middleware based on the token bucket algorithm.
Password hashing storage and verification.
Detailed login history records.
Unified error handling middleware.
Supports HTTPS (can be enabled in configuration).
The layered architecture encapsulates security mechanisms uniformly in the infrastructure layer, facilitating unified management and maintenance.

📊 Observability

Integrated structured logging system (zap).
Health check interface for monitoring system status.
Detailed request/response logging.
Supports custom monitoring metrics.
The layered architecture independently encapsulates monitoring functions, ensuring observability of the operational status of each system layer.
Integrated Prometheus and Grafana monitoring system providing visual dashboards.
Supports custom alert rule configuration.

🚀 Developer Friendly

Complete plugin development documentation and examples.
Plugin scaffolding tool for quickly generating plugin templates.
Supports local development and Docker deployment.
Clear project structure and code specifications.

📂 Project Structure

Weave adopts a microkernel + layered architecture, and the project structure clearly reflects this design philosophy. The core system is organized in layers, while functional extensions are achieved through the plugin mechanism/service aggregation.

├── .github/
├── .gitignore
├── Dockerfile           # Docker build file
├── Makefile             # Build scripts
├── README.md
├── config/              # Configuration management
├── controllers/         # API controllers [Interface Layer]
├── docker-compose.yaml  # Docker Compose configuration
├── docs/                # Project documentation
├── go.mod
├── go.sum
├── main.go
├── middleware/          # Middleware
├── models/              # Data models [Data Layer]
├── pkg/                 # Common packages [Infrastructure Layer]
├── plugins/             # Plugin system [Core of Microkernel Architecture]
│   ├── core/              # Core plugin functionality
│   ├── doc.go             # Plugin package documentation
│   ├── examples/          # Example plugins
│   ├── features/          # Feature plugins (extensible)
│   ├── init.go            # Plugin initialization
│   ├── loader/            # Plugin loader
│   ├── templates/         # Plugin templates
│   └── watcher/           # Plugin watcher
├── routers/             # Route definition and registration
├── services/            # Service aggregation
│   ├── llm/               # LLM Service
│   ├── rag/               # RAG Service  
│   └── extended/          # Extensible services
├── test/                # Unit/Integration tests
├── tools/               # Development tools
├── utils/               # Utility functions
└── web/                 # Frontend code

🧩 Core Components

🔌 Plugin System - Core Implementation of Microkernel Architecture

The plugin system is an important component of Weave, responsible for plugin registration, loading, unloading, and lifecycle management. It implements a complete plugin mechanism, enabling the system to extend functionality in the form of plugins. In the microkernel + layered architecture, the plugin system connects the core kernel with various business extensions.

Under the microkernel architecture, the plugin system has the following characteristics:

Complete Lifecycle Management: Full lifecycle management from plugin initialization, registration, activation to shutdown.
Automatic Dependency Resolution: Automatically resolves dependencies between plugins via the GetDependencies() method.
Conflict Detection Mechanism: Avoids functional conflicts between plugins via the GetConflicts() method.
Automatic Route Registration: Supports two methods of route registration; the recommended GetRoutes() method aligns better with the microkernel architecture design philosophy.
Namespace Isolation: Each plugin has an independent namespace to avoid resource conflicts.
Unified Middleware Management: Supports global and plugin-level middleware configuration.

// Plugin interface definition
type Plugin interface {
    Name() string              // Plugin name
    Description() string       // Plugin description
    Version() string           // Plugin version
    Init() error               // Initialize plugin
    Shutdown() error           // Shutdown plugin
    
    // Route management (new way) - Recommended
    GetRoutes() []Route
    GetDefaultMiddlewares() []gin.HandlerFunc
    
    // Route management (old way) - Retained for compatibility
    RegisterRoutes(*gin.Engine) // Register routes
    
    Execute(map[string]interface{}) (interface{}, error) // Execute function
}

// The Route struct defines route metadata and handler functions.
// This is the core of the new route definition method.
type Route struct {
    Path         string                 // Route path
    Method       string                 // HTTP method (GET, POST, PUT, DELETE, etc.)
    Handler      gin.HandlerFunc        // Request handler function
    Middlewares  []gin.HandlerFunc      // Route-specific middlewares
    Description  string                 // Route description
    AuthRequired bool                   // Whether authentication is required
    Tags         []string               // Route tags, for documentation generation
    Params       map[string]string      // Parameter descriptions, for documentation generation
    Metadata     map[string]interface{} // Custom metadata
}

The plugin manager is responsible for the entire lifecycle management of plugins, including registration, deregistration, querying, and executing plugin functions.

🧩 Service Aggregation

Service aggregation is an important extension capability of Weave based on the microkernel + layered architecture, providing a mechanism for unified management and invocation of various services, data sources, and functions. For example, the LLM-RAG service aggregation.

LLM-RAG Retrieval-Augmented Generation Service, as one of Weave's service aggregations, provides intelligent text retrieval and enhanced generation functions.
Efficient Vector Retrieval: High-performance vector similarity search based on RedisSearch.
Multi-format Document Support: Supports parsing, chunking, and vectorization of various document formats.
Flexible Retrieval API: Provides rich retrieval interfaces supporting multiple retrieval strategies.
Configurable Embedding Models: Supports switching different embedding models to adapt to different scenario requirements.
Integration with Plugin System: Can be used as infrastructure called by various plugins, enhancing plugin intelligence capabilities.
Independent Deployment Option: Supports running as an independent service or integrating into the main application.

The service aggregation design enhances system functional flexibility, allowing the system to extend and integrate various services and data sources, providing stronger underlying capability support.

🔐 Authentication System

The authentication system is located in the infrastructure layer of the layered architecture, providing comprehensive identity authentication and authorization mechanisms, supporting multiple authentication methods. The authentication system is closely integrated with the plugin system, ensuring secure access to plugins, while achieving unified management of security mechanisms through layered design.

JWT-based token authentication.
Supports access token and refresh token mechanisms.
Password hashing storage enhances security.
Login history records facilitate auditing and tracking.
Role-based access control.

🔄 Middleware System

The middleware system is located between the interface layer and the business layer of the layered architecture, supporting global middleware and plugin-level middleware. It can be used for scenarios such as logging, request validation, and performance monitoring. The middleware system adopts a chain invocation pattern, flexibly combining various functions, reflecting the request processing optimization of the layered architecture.

Authentication Middleware: Verifies user identity.
Rate Limiting Middleware: Prevents API abuse.
CORS Middleware: Handles cross-origin requests.
CSRF Protection Middleware: Prevents cross-site request forgery.
Error Handling Middleware: Uniformly handles and logs errors.

📈 Monitoring System

Weave integrates a complete Prometheus + Grafana monitoring system:

Automatically collects application runtime metrics.
Pre-configured with various visualization dashboards.
Supports custom alert rules.
Real-time monitoring of system health status and performance metrics.

🩺 Health Check

The health check function covers all layers of the layered architecture, periodically checking the operational status of various system components to ensure stable system operation. Supports custom health check items to meet the needs of different scenarios. Through the microkernel + layered architecture design, health checks can precisely target the operational status of each plugin and each layer.

Database connection health check.
Plugin system status check.
Overall system health assessment.
Returns appropriate HTTP status codes based on health status.

Quick Start

Environment Preparation

Go 1.21+ (for local development)
Git (for cloning the repository)
Docker and Docker Compose (for containerized deployment)
MySQL 8.0+
PostgreSQL、Redis、Prometheus、Grafana (Optional, for extension)

Deployment Methods

1. Docker Compose Deployment (Recommended)

Clone the repository

git clone https://github.com/liaotxcn/weave.git
cd weave

Create an environment variable file (Optional but recommended) Create a .env file to set environment variables for enhanced security.
Start the services Use Docker Compose to start the entire service stack with one command:

docker-compose up -d

On the first startup, Docker Compose will automatically:

Build the Docker image for the Weave application.
Create the MySQL database container.
Create the RedisSearch vector database container.
Configure the Prometheus and Grafana monitoring system.
Configure networks and volumes.
Start all services.

After the services start, you can access the following addresses:

Application Backend: http://localhost:8081
Prometheus Monitoring: http://localhost:9090
Grafana Dashboard: http://localhost:3000 (Default credentials: admin/admin)

Verify Service Status Check if all services are running normally:

docker-compose ps

Under normal circumstances,weave-app、weave-mysqlandweave-redis should all show Up status.

Docker Compose Commands

docker-compose down    // Stop services
docker-compose logs -f weave-app   // View application logs
docker-compose logs -f weave-mysql // View database logs
docker-compose logs -f weave-redis    // View Redis logs
docker-compose exec weave-app /bin/sh             // Enter the application container
docker-compose exec weave-mysql mysql -u root -p  // Enter the database container
docker-compose exec weave-redis redis-cli    // Enter the Redis container
docker-compose up --build -d        // Rebuild and start services

// Clean up old containers and volume data
docker-compose down -v 
docker system prune -f
docker-compose build --no-cache     // Rebuild images
docker-compose up --force-recreate -d   // Start with --force-recreate option

2. Local Development Environment Setup

Clone the repository and enter the project directory

git clone https://github.com/liaotxcn/weave.git
cd weave

Install dependencies

go mod download

Configure the database Ensure the local MySQL service is started and create the database:

CREATE DATABASE weave;

Set environment variables or modify the default configuration in config/config.go
Run the application

go run main.go

Build the application

go build

Frontend Build

cd web
npm install
npm run dev

Notes

Data Persistence:
- MySQL data is stored in the mysql-data volume, ensuring data is not lost
- RedisSearch data is stored in the redis-data volume, ensuring vector index data is not lost
Health Check: The system provides a /health interface to monitor service health status
Resource Limits: CPU and memory limits are configured by default and can be adjusted in docker-compose.yaml according to actual needs
First Startup: The first startup requires some time to build images and initialize services
Port Mapping:
- By default, the container's port 8081 is mapped to the host's port 8081
- By default, the container's port 6379 is mapped to the host's port 6379 (RedisSearch)

Project Documentation

Please read in detail

API Documentation Plugin Development Guide Plugin Scaffold Tool Usage Database Migration Guide Monitoring System Guide

🔧 Creating a New Plugin

In Weave's microkernel + layered architecture, creating a new plugin is one way to extend system functionality. A plugin is a Go struct that implements the Plugin interface. Through this interface, the plugin can interact with the core system. The microkernel architecture provides plugin flexibility, while the layered architecture provides good guidance for the internal code organization of the plugin.

Creating a new plugin is very efficient, just follow these steps:

Implement the plugins.Plugin interface, defining the plugin's basic information, lifecycle, and functionality
Register the plugin in the registerPlugins function in main.go

Advantages of plugin development in the microkernel + layered architecture:

Low Intrusiveness: Extend system functionality without modifying core code
Independent Evolution: Plugins can be developed, tested, and deployed independently
Standardized Interface: Unified plugin interface simplifies the development process
Flexible Combination: Users can combine different plugins according to their needs
Clear Structure: The layered architecture philosophy guides the internal code organization of plugins, improving maintainability

Plugin Example (Using the recommended GetRoutes method)

// Example plugin structure
type MyPlugin struct{}

// Methods implementing the Plugin interface
func (p *MyPlugin) Name() string { return "myplugin" }
func (p *MyPlugin) Description() string { return "My custom plugin" }
func (p *MyPlugin) Version() string { return "1.0.0" }
func (p *MyPlugin) Init() error { /* Initialization logic */ return nil }
func (p *MyPlugin) Shutdown() error { /* Shutdown logic */ return nil }

// Register routes using the recommended GetRoutes method
func (p *MyPlugin) GetRoutes() []Route {
    return []Route{
        {
            Path:        "/",
            Method:      "GET",
            Handler:     p.handleIndex,
            Description: "Plugin homepage",
            AuthRequired: false,
            Tags:        []string{"home"},
        },
        {
            Path:        "/api/data",
            Method:      "GET",
            Handler:     p.handleGetData,
            Description: "Get data API",
            AuthRequired: true,
            Tags:        []string{"data", "api"},
            Params: map[string]string{
                "id": "Data ID",
            },
        },
    }
}

// Define the plugin's default middlewares
func (p *MyPlugin) GetDefaultMiddlewares() []gin.HandlerFunc {
    return []gin.HandlerFunc{
        p.logMiddleware,
    }
}

// Route handler functions
func (p *MyPlugin) handleIndex(c *gin.Context) {
    c.JSON(200, gin.H{
        "plugin": p.Name(),
        "version": p.Version(),
    })
}

func (p *MyPlugin) handleGetData(c *gin.Context) {
    id := c.Query("id")
    c.JSON(200, gin.H{
        "id": id,
        "data": "Example data",
    })
}

// Middleware example
func (p *MyPlugin) logMiddleware(c *gin.Context) {
    // Log request
    c.Next()
}

// RegisterRoutes method retained for compatibility
func (p *MyPlugin) RegisterRoutes(router *gin.Engine) {
    // Note: Using GetRoutes method is recommended, this method is only retained for compatibility
    // Can keep empty implementation or add log hint here
}

// Plugin execution logic
func (p *MyPlugin) Execute(params map[string]interface{}) (interface{}, error) {
    // Implement plugin functionality
    return map[string]interface{}{"result": "success"}, nil
}

Plugin Example (RegisterRoutes method - Retained only for compatibility)

// Register plugin routes
func (p *MyPlugin) RegisterRoutes(router *gin.Engine) {
    group := router.Group(fmt.Sprintf("/plugins/%s", p.Name()))
    {
        group.GET("/", func(c *gin.Context) {
            c.JSON(200, gin.H{"plugin": p.Name()})
        })
        // Add more routes...
    }
}

Comparison of Two Route Registration Methods

Feature	GetRoutes Method (Recommended)	RegisterRoutes Method (Compatibility)
Route Definition	Uses Route struct array	Directly operates gin.Engine object
Metadata Support	✅ Full support	❌ Not supported
Automatic Route Group	✅ Automatically created	❌ Requires manual creation
Middleware Management	✅ Supports global and route level	❌ Requires manual addition
Documentation Generation	✅ Supports automatic API doc generation	❌ Not supported

📊 Database Migration Tool

Weave provides an efficient and powerful database migration tool located in the pkg/migrate directory, supporting version management of the database structure:

Implemented based on the golang-migrate library
Supports migration application, rollback, status query, etc.
Automatically generates version numbers to avoid conflicts
Supports migration status check and dirty state handling

🤝 Contribution Guide

Welcome contributions to the project! Thank you!

Fork the repository and clone it locally
Create a branch for development (git checkout -b feature/your-feature)
Commit your code and ensure tests pass
Create a Pull Request describing your changes
Wait for code review and make modifications based on feedback

Name		Name	Last commit message	Last commit date
Latest commit History 172 Commits
.github/workflows		.github/workflows
config		config
controllers		controllers
docs		docs
middleware		middleware
models		models
pkg		pkg
plugins		plugins
routers		routers
services		services
test		test
tools		tools
utils		utils
web		web
.gitignore		.gitignore
Dockerfile		Dockerfile
Makefile		Makefile
README.md		README.md
README.zh-CN.md		README.zh-CN.md
docker-compose.yaml		docker-compose.yaml
go.mod		go.mod
go.sum		go.sum
main.go		main.go

liaotxcn/Weave

Folders and files

Latest commit

History

Repository files navigation