🗄️ nexDB

A minimal, lightweight, and educational C-based key-value NoSQL database with TCP interface, automatic sharding, and Redis-like command set

Note: This is an educational project designed for learning database internals, prototyping, and resource-constrained environments.

---

📋 Table of Contents

• Features
• Requirements
• Installation
• Quick Start
• Configuration
• Command Reference
• Usage Examples
• Architecture
• Logging
• Contributing
• License

---

✨ Features

• 🔑 Key-Value Storage - Simple and fast key-value operations with automatic sharding
• 🌐 TCP Server Interface - Network-accessible database server with authentication
• 🔐 Authentication System - SHA-256 password hashing with role-based access control (read/write permissions)
• 📦 Automatic Sharding - Key space automatically distributed across multiple files for better performance
• ⏱️ TTL Support - Time-to-live expiration for keys with multiple time formats
• 📊 JSON Operations - Built-in JSON parsing and manipulation with path-based access
• 🔢 Numeric Operations - Arithmetic operations, increments, and mathematical functions
• 📝 String Manipulation - Rich set of string operations (append, prepend, trim, case conversion, etc.)
• 🔍 Pattern Matching - Regex support and prefix-based scanning
• 🎯 Bit Operations - Bit manipulation commands for advanced use cases
• 📈 Multi-Key Operations - Batch operations for improved efficiency
• 📋 Collection API - Helper API for fixed-size record storage across multiple files
• 🚀 Zero Runtime Dependencies - Standalone binary with minimal build dependencies
• 📝 Comprehensive Logging - Detailed logs for authentication, operations, errors, and debugging

---

📦 Requirements

• C Compiler: GCC or Clang supporting C11 standard
• CMake: Version 3.10 or higher
• libconfig: For configuration file parsing
• OpenSSL: For SHA-256 password hashing
• POSIX-compliant OS: Linux, macOS, or similar Unix-like system

---

🔧 Installation

Building from Source

1. Clone the repository:

   git clone <repository-url>
   cd nexDB-source

2. Build the project:

   cmake -S . -B build && cmake --build build

3. Find the executable: The nexDb binary will be located in the build/ directory.

Dependencies

Install required dependencies:

Ubuntu/Debian:

sudo apt-get install libconfig-dev libssl-dev cmake build-essential

macOS:

brew install libconfig openssl cmake

Fedora/RHEL:

sudo dnf install libconfig-devel openssl-devel cmake gcc

---

🚀 Quick Start

1. Configure Authentication

Edit auth_configuration.conf to set up users. Generate a password hash:

printf 'your_password' | sha256sum

Add the hash to the configuration:

users = (
    {
        id = "admin";
        pass = "<your_sha256_hash>";
        access = "rw";  # "r" for read-only, "rw" for read-write
    }
);

2. Configure Host Settings

Edit host_configuration.conf:

host_name = "0.0.0.0"      # Listen on all interfaces
host_port = 5555           # Port number
max_client_connections = 100

3. Start the Server

./build/nexDb

The server will start and listen on the configured address and port.

4. Connect and Use

Using netcat or telnet:

# Authenticate and store a value
printf "AUTH admin your_password\nPUT name Alice\n" | nc localhost 5555

# Retrieve the value
printf "AUTH admin your_password\nGET name\n" | nc localhost 5555

Output:

OK
Alice

---

⚙️ Configuration

Authentication Configuration (auth_configuration.conf)

users = (
    {
        id = "username";
        pass = "sha256_hash";
        access = "rw";  # "r" or "rw"
    }
);

• id: Username for authentication
• pass: SHA-256 hash of the password
• access: Permission level (r for read-only, rw for read-write)

Host Configuration (host_configuration.conf)

host_name = "0.0.0.0"
host_port = 5555
max_client_connections = 100

• host_name: IP address to bind to (0.0.0.0 for all interfaces)
• host_port: TCP port number
• max_client_connections: Maximum concurrent client connections

Socket Configuration (socket_configuration.conf)

tcp_no_delay = true

• tcp_no_delay: Enable/disable TCP_NODELAY option

---

📖 Command Reference

🔑 Basic Key-Value Operations

Command	Description	Permission	Example
PUT <key> <value>	Store a value by key	w	PUT name Alice
GET <key>	Retrieve a value by key	r	GET name
DEL <key>	Delete a key	w	DEL name
EXISTS <key>	Check if key exists (returns 1 or 0)	r	EXISTS name
RENAME <old> <new>	Rename a key	w	RENAME old new
RENAMENX <old> <new>	Rename only if destination doesn't exist	w	RENAMENX old new
COPY <src> <dst>	Copy value to a new key	w	COPY src dst
SWAP <key1> <key2>	Swap values between two keys	w	SWAP a b
GETSET <key> <value>	Set value and return previous (or NIL)	w	GETSET counter 5
GETDEL <key>	Get value and delete key	w	GETDEL temp
SETNX <key> <value>	Set only if key doesn't exist (returns 1 or 0)	w	SETNX new 1

📊 JSON Operations

Command	Description	Permission	Example
JSONPUT <key> JSON_START ... JSON_END	Store multi-line JSON value	w	JSONPUT doc JSON_START\n{"name":"Bob"}\nJSON_END
JSONGET <key> <path>	Get JSON field by path (supports dotted paths)	r	JSONGET user score
JSONSET <key> <path> <json>	Set or replace JSON field	w	JSONSET user level 5
JSONDEL <key> <path>	Delete JSON field	w	JSONDEL user score
JSONGETBEST <limit> <prefix> <path>	Get top N keys by JSON numeric value	r	JSONGETBEST 10 user: score
JSONGETWORST <limit> <prefix> <path>	Get bottom N keys by JSON numeric value	r	JSONGETWORST 10 user: score

📝 String Operations

Command	Description	Permission	Example
APPEND <key> <suffix>	Append text to value	w	APPEND name Smith
PREPEND <key> <prefix>	Prepend text to value	w	PREPEND name Dr.
STRLEN <key>	Get string length (or -1 if missing)	r	STRLEN name
GETRANGE <key> <start> <end>	Get substring by index range	r	GETRANGE text 0 4
SETRANGE <key> <offset> <value>	Overwrite part of string	w	SETRANGE text 5 XYZ
TRIM <key> <length>	Truncate string to length	w	TRIM text 10
LPAD <key> <length> [pad]	Left pad string (default: space)	w	LPAD num 5 0
RPAD <key> <length> [pad]	Right pad string (default: space)	w	RPAD num 5 0
UPPER <key>	Convert to uppercase	w	UPPER text
LOWER <key>	Convert to lowercase	w	LOWER text
LSTRIP <key>	Remove leading whitespace	w	LSTRIP text
RSTRIP <key>	Remove trailing whitespace	w	RSTRIP text
STRIP <key>	Remove leading and trailing whitespace	w	STRIP text
REVERSE <key>	Reverse the string	w	REVERSE text
REPEAT <key> <count>	Repeat string N times	w	REPEAT hi 3
POP <key>	Remove and return last character	w	POP text
REPLACE <key> <old> <new>	Replace first occurrence	w	REPLACE text old new
STRPOS <key> <substr>	Find substring position (or -1)	r	STRPOS text hello
GETREGEX <key> <pattern>	Get first regex match	r	GETREGEX text [0-9]+
SETREGEX <key> <pattern> <replacement>	Replace first regex match	w	SETREGEX text \d+ 999

🔢 Numeric Operations

Command	Description	Permission	Example
INCR <key>	Increment by 1	w	INCR counter
INCRBY <key> <n>	Increment by N	w	INCRBY counter 5
INCRBYFLOAT <key> <n>	Increment by floating-point N	w	INCRBYFLOAT price 1.5
DECR <key>	Decrement by 1	w	DECR counter
DECRBY <key> <n>	Decrement by N	w	DECRBY counter 3
MULBY <key> <n>	Multiply value by N	w	MULBY num 5
DIVBY <key> <n>	Divide value by N (integer division)	w	DIVBY num 2
MODBY <key> <n>	Modulo operation (value % N)	w	MODBY num 4
POWBY <key> <n>	Raise value to Nth power	w	POWBY num 8
ABS <key>	Absolute value	w	ABS num
EVAL <key>	Evaluate arithmetic expression stored as value	r	EVAL expr

⏱️ Expiration & TTL Operations

Command	Description	Permission	Example
PUTEX <key> <seconds> <value>	Store with TTL in seconds	w	PUTEX temp 60 value
EXPIRE <key> <seconds>	Set TTL in seconds	w	EXPIRE key 300
PEXPIRE <key> <ms>	Set TTL in milliseconds	w	PEXPIRE key 5000
EXPIREAT <key> <unix>	Set expiration using Unix timestamp	w	EXPIREAT key 1700000000
TTL <key>	Get remaining TTL in seconds (-1 persistent, -2 missing)	r	TTL key
PTTL <key>	Get remaining TTL in milliseconds	r	PTTL key
EXPIRETIME <key>	Get absolute expiration Unix timestamp	r	EXPIRETIME key
PEXPIRETIME <key>	Get absolute expiration in milliseconds	r	PEXPIRETIME key
PERSIST <key>	Remove TTL (make persistent)	w	PERSIST key
GETEX <key> <seconds>	Get value and set new TTL	w	GETEX temp 30

📦 Multi-Key Operations

Command	Description	Permission	Example
MSET <key> <value> [key value ...]	Set multiple keys	w	MSET a 1 b 2 c 3
MGET <key> [key ...]	Get multiple keys (returns NIL for missing, ends with END)	r	MGET a b c
MSETNX <key> <value> [key value ...]	Set multiple keys only if none exist	w	MSETNX a 1 b 2
MDEL <key> [key ...]	Delete multiple keys (returns count)	w	MDEL a b c

🔍 Key Scanning & Management

Command	Description	Permission	Example
LIST	List all key-value pairs (ends with END)	r	LIST
SCAN <prefix>	List keys starting with prefix (ends with END)	r	SCAN user:
KEYS <prefix>	List all keys with prefix (ends with END)	r	KEYS user:
COUNT [prefix]	Count total keys or keys with prefix	r	COUNT user:
DBSIZE	Get total number of keys	r	DBSIZE
RANDOMKEY	Get random key (or NIL if empty)	r	RANDOMKEY
RANDOM	Get random key-value pair (or NIL)	r	RANDOM
DELPREFIX <prefix>	Delete all keys with prefix (returns count)	w	DELPREFIX temp:
DELIF <key> <value>	Delete only if value matches (returns 1 or 0)	w	DELIF key value

🎯 Bit Operations

Command	Description	Permission	Example
SETBIT <key> <offset> <value>	Set/clear bit at offset (returns previous bit)	w	SETBIT flag 0 1
GETBIT <key> <offset>	Get bit at offset (or 0 if out of range)	r	GETBIT flag 0
BITCOUNT <key>	Count number of set bits	r	BITCOUNT flag

🗄️ Database Management

Command	Description	Permission	Example
FLUSH	Delete all keys	w	FLUSH
SHARDSIZE <index>	Get size of shard file in bytes	r	SHARDSIZE 0
TOTALSIZE	Get total size of all shards in bytes	r	TOTALSIZE
QUIT	Close connection	-	QUIT

---

📖 Usage Examples

Example 1: Basic Operations

# Store and retrieve values
printf "AUTH admin password\nPUT name Alice\nGET name\n" | nc localhost 5555

Output:

OK
OK
Alice

Example 2: JSON Operations

# Store JSON and access fields
printf "AUTH admin password\nPUT player {\"score\":10,\"name\":\"Bob\"}\nJSONGET player score\nJSONSET player level 5\nJSONGET player level\n" | nc localhost 5555

Output:

OK
OK
10
OK
5

Example 3: Multi-line JSON

# Store multi-line JSON
printf "AUTH admin password\nJSONPUT doc JSON_START\n{\n  \"name\": \"User3\",\n  \"score\": 99\n}\nJSON_END\nJSONGET doc score\n" | nc localhost 5555

Output:

OK
OK
99

Example 4: TTL and Expiration

# Store with expiration
printf "AUTH admin password\nPUTEX temp 5 value\nTTL temp\n" | nc localhost 5555

# Wait 6 seconds, then check
sleep 6
printf "AUTH admin password\nGET temp\n" | nc localhost 5555

Output:

OK
5
ERR  # After expiration

Example 5: String Manipulation

printf "AUTH admin password\nPUT text hello\nAPPEND text world\nUPPER text\nREVERSE text\n" | nc localhost 5555

Output:

OK
helloworld
HELLOWORLD
DLROWWOLLEH

Example 6: Numeric Operations

printf "AUTH admin password\nPUT counter 0\nINCR counter\nINCRBY counter 5\nMULBY counter 2\nGET counter\n" | nc localhost 5555

Output:

OK
1
6
12
12

Example 7: Pattern Matching and Scanning

# Store multiple keys with prefix
printf "AUTH admin password\nPUT user:1 Bob\nPUT user:2 Alice\nSCAN user:\n" | nc -N localhost 5555

Output:

OK
OK
user:1=Bob
user:2=Alice
END

Example 8: JSON Ranking

# Find top scores
printf "AUTH admin password\nPUT u:1 {\"score\":15}\nPUT u:2 {\"score\":25}\nJSONGETBEST 2 u: score\n" | nc localhost 5555

Output:

OK
OK
u:2 25
u:1 15
END

---

🏗️ Architecture

Storage System

• Sharded Storage: Keys are automatically distributed across multiple data.X.db files using hash-based sharding
• Binary Format: Efficient binary storage for fast reads and writes
• Collection API: Additional API (collection_dist) for fixed-size record storage across multiple files

Server Architecture

• TCP Server: Multi-process server that forks a new process for each client connection
• Authentication: SHA-256 password hashing with role-based access control
• Command Processing: Line-based protocol similar to Redis
• Concurrent Access: Multiple clients can connect simultaneously

Key Components

• kv_store: Core key-value storage engine
• kv_store_dist: Distributed/sharded storage layer
• collection: Fixed-size record collection API
• collection_dist: Distributed collection storage
• server: TCP server implementation
• auth_configuration: User authentication management
• host_configuration: Network configuration
• socket_configuration: Socket options
• logger: Comprehensive logging system

---

📝 Logging

The server generates detailed logs in the working directory:

Log File	Purpose
auth.log	Connection attempts, authentication results, timestamps, and client addresses
operations.log	Every command executed by authenticated users
error.log	Internal errors and unknown commands
debug.log	Debugging information, server start messages, and miscellaneous details

Note: Log files grow over time and should be rotated or cleaned periodically.

---

🤝 Contributing

Contributions are welcome! This is an educational project, so contributions that improve documentation, add examples, or enhance educational value are especially appreciated.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

🙏 Acknowledgments

• Built with pure C for minimal dependencies
• Uses libconfig for configuration parsing
• Uses OpenSSL for cryptographic operations
• Inspired by Redis and other key-value stores
• Designed for educational purposes and learning database internals

---

📞 Support

If you encounter any issues or have questions, please open an issue on the GitHub repository.

---

Made with ❤️ by neziw for learning and education