MYSH Unix Shell

POSIX-Style Command Line Shell Built in C

Summary

MYSH is a compact POSIX-style Unix command-line shell implemented in C. It provides a functional set of shell capabilities including command parsing, pipelines, input/output redirection, job control, and builtin commands.

The shell was designed to demonstrate core systems programming concepts such as process creation, signal handling, process groups, and file descriptor management. MYSH closely follows the behavior of traditional Unix shells, making it useful both as an educational tool and as a practical demonstration of low-level operating system concepts.

Key Features

• Command Parsing with tokenization and basic quoting support.
• Pipelines supporting multi-stage command chaining using the pipe operator |.
• Redirection support including <, >, >>, numeric file descriptor redirection (2>file) and descriptor duplication (2>&1).
• Job Control for background processes using & with job tracking and control commands.
• Built-in Commands including cd, pwd, exit, echo, alias, unalias, type, help, export, set, unset, and history.
• Execution Engine implementing fork/exec with correct process-group handling and foreground/background execution semantics.

Design & Architecture

MYSH follows a classic Unix shell architecture with modular components responsible for parsing commands, executing processes, and managing jobs.

• REPL Layer collects logical command lines and supports multi-line quoted inputs.
• Tokenizer & Parser convert raw input into a small command pipeline AST.
• Executor manages pipes, process creation, file descriptor redirection, and builtin execution.
• Job Control System tracks background jobs and safely reaps child processes using SIGCHLD.

Runtime Execution Flow

1. Read input from the user and construct a logical command line.
2. Tokenize the command while respecting quoted strings and escape sequences.
3. Build pipeline structures where each stage represents a command execution unit.
4. Create pipes, fork processes, configure redirections, and execute commands.
5. Parent process manages foreground/background execution and updates the job table.

Core Data Model

Internally, MYSH relies on structured representations for commands and job management.

• command_t — represents a pipeline stage containing arguments, redirection settings, and pipeline links.
• job_t — tracks job metadata including job ID, process group ID, command string, child PIDs, state, and exit status.

Built-in Commands

MYSH implements several built-in commands commonly found in Unix shells. Commands that modify shell state such as cd or export are executed in the parent process, while other builtins can run inside pipeline child processes when required.

Redirections are applied transparently so builtins behave consistently with external commands.

Example Commands

# basic command
echo hello

# pipeline example
/usr/bin/printf "a\nb\nc\n" | /usr/bin/grep b | /usr/bin/wc -l

# file redirection
echo redir_test > /tmp/mysh_test1

# background job
sleep 10 &
jobs
fg %1

Build & Execution

# build project
make

# start interactive shell
./bin/mysh

# run scripted commands
printf 'echo hello' | ./bin/mysh

Future Improvements

• Enhanced variable and word expansion.
• Persistent command history with line editing.
• Support for here-documents.
• Improved terminal foreground control using tcsetpgrp.

Detailed documentation, build instructions, and technical examples are provided in the project documentation.

DocumentBot

Enterprise Information Retrieval & AI Question Answering Platform

Summary

DocumentBot is an enterprise-grade information retrieval and question-answering system designed to help users search, explore, and extract answers from large collections of documents including PDFs, Word files, HTML pages, and scanned documents.

The platform combines traditional lexical search techniques such as BM25, TF-IDF, Boolean search, and fuzzy matching with modern dense retrieval using semantic embeddings and vector similarity search. Optional neural rerankers and Retrieval-Augmented Generation (RAG) allow the system to generate contextual answers with citations.

Users can explicitly choose their preferred retrieval strategy or allow the system to dynamically select a hybrid retrieval method that maximizes both precision and recall.

Real-World Use Cases

• Support Knowledge Base — helpdesk agents instantly locate procedures, troubleshooting steps, and code snippets within large product manuals.
• Legal Discovery — identify relevant clauses across contracts using boolean filters and fuzzy matching combined with semantic search.
• Research Assistant — extract summaries, citations, and passages from research papers for literature reviews.
• Enterprise Document Search — perform unified semantic and metadata search across manuals, SOPs, incident logs, and internal documentation.

Key Platform Features

• Multi-Method Retrieval — lexical search (BM25, TF-IDF), boolean queries, fuzzy search, and semantic embedding-based retrieval.
• Vector Indexing using high-performance vector databases such as FAISS, Milvus, and Pinecone with ANN indexing strategies.
• Hybrid Search Engine combining lexical and semantic similarity scores with configurable weighting.
• Neural Reranking using cross-encoder models to reorder top candidate results for higher precision.
• Answer Extraction & RAG retrieving relevant passages and generating concise answers with source citations.
• Query Intelligence including autocomplete, spell correction, synonym expansion, and query suggestions.
• User Profiles & Retrieval Modes allowing users to select fast, semantic, hybrid, or advanced boolean search modes.
• Robust Document Ingestion pipeline supporting PDFs, DOCX, HTML, and OCR for scanned documents.
• Monitoring & Analytics including query latency tracking, search analytics, and relevance feedback loops.

Algorithms & Retrieval Methods

The platform integrates a wide spectrum of classical and modern information retrieval techniques.

• Lexical Retrieval — TF-IDF vectorization, BM25 ranking, boolean queries, and fuzzy matching using edit distance.
• Dense Retrieval — sentence and paragraph embeddings using SBERT and sentence-transformer models.
• Approximate Nearest Neighbor Search using HNSW, IVF, and PQ indexing techniques.
• Neural Rerankers — cross-encoder models improving ranking precision for top candidate passages.
• Hybrid Ranking combining lexical and semantic scores using configurable weighting strategies.
• Answer Extraction — deterministic span extraction or LLM-generated responses with provenance citations.
• Query Expansion through synonym dictionaries, pseudo-relevance feedback, and semantic expansion.

System Architecture

DocumentBot follows a modular pipeline architecture that separates ingestion, indexing, retrieval, and serving components.

1. Ingestion — extract text and metadata from PDFs, DOCX, and HTML documents with optional OCR for scanned files.
2. Indexing — build both lexical indexes (Elasticsearch/Whoosh) and dense vector indexes (FAISS/Milvus/Pinecone).
3. Query Processing — normalize and preprocess queries before executing lexical and dense retrieval methods in parallel.
4. Ranking & Extraction — merge results, optionally rerank using neural models, and extract answer passages.
5. Serving Layer — REST APIs and interactive UI displaying highlighted snippets with source document references.

Indexing Strategy

• Documents are split into 200–800 token chunks with overlapping context windows.
• Both paragraph-level and sliding-window chunk representations are stored for better semantic coverage.
• Metadata fields such as title, author, date, and tags are indexed for filtering and faceted search.

User Experience

• Users can search using natural language questions.
• Select retrieval mode: Fast (Lexical), Semantic, Hybrid, or Advanced Boolean.
• View highlighted passages with relevance scores and citations linking back to the source document.
• Provide feedback to improve future ranking performance.

Example Query

Q: How do I reset the device to factory settings?

Top Result:
"To perform a factory reset, press and hold the reset button for 10 seconds..."

Source: manual_v2.pdf — Page 12
Score: 0.92

Evaluation & Metrics

• Offline metrics such as MRR, nDCG@K, and Precision@K on annotated evaluation datasets.
• Online monitoring including click-through rate, user satisfaction signals, and latency metrics.
• A/B testing comparing hybrid ranking against purely lexical retrieval baselines.

Build & Run

# Setup environment
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# Run ingestion pipeline
python ingest.py --source ./data/manuals --index dense --chunksize 500

# Start API server
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

Production Considerations

• Vector databases can scale independently using sharding and replicas.
• Lexical search results can be cached to reduce query latency.
• Secure authentication and role-based access control protect private document collections.
• Access logs and provenance tracking support compliance and auditing requirements.

Future Improvements

• Multilingual embeddings and cross-language retrieval.
• Online learning using user feedback to optimize ranking models.
• Automatic summarization and exportable citations for downstream workflows.

DocumentBot is designed as a modular, production-ready information retrieval platform suitable for enterprise search, legal discovery, compliance analysis, and support automation systems.