Set up, optimize, and verify a reliable local Qwen3.8 environment on this Windows computer. HARDWARE - Operating system: Windows - GPU: NVIDIA GeForce RTX 3090 - Dedicated VRAM: 24 GB - Typical free VRAM before loading the model: approximately 22 GB - System RAM: 128 GB - Typical available system RAM: approximately 106 GB RAM AND VRAM SAFETY The Q4_K_M weights should primarily occupy GPU VRAM. System RAM may also show a memory-mapped or cached copy of the model; cached memory is reclaimable and does not mean Windows has permanently lost that RAM. Treat excessive RAM or VRAM use as a performance/stability concern, not a hardware-damage risk. If memory pressure becomes excessive, stop the model server cleanly; do not allow repeated out-of-memory restart loops. During every context test, record: - Dedicated VRAM used and available - System RAM in use and available - Whether model layers, KV cache, or compute buffers spilled into system RAM - Whether Windows began paging Maintain at least 2 GB of practical VRAM headroom during the stable everyday profile and substantial system-RAM headroom for Windows and development tools. Do not configure an 80 GB-class RAM-resident model as the default while also claiming this Q4 profile is lightweight. INTENDED USE The model will be served locally through an OpenAI-compatible endpoint and used by: - Hermes Agent as the primary rich agent harness - Pi or another minimal coding harness - Potentially other local clients later The model files must remain independent of any individual harness. Maintain one canonical model directory and avoid unnecessary duplicate weight downloads. AUTONOMY TARGET Hermes should eventually operate with high autonomy for ordinary local work: - Read and edit authorized project files - Run normal development commands - Use browser and computer tools - Install normal project dependencies - Iterate and verify without repeatedly requesting approval Do not weaken operating-system security, expose the inference server publicly, delete existing configurations, or disable Hermes’s irreducible catastrophic-command protections. EPISTEMIC AND FACT-CHECKING REQUIREMENT Treat every instruction and assumption in this prompt as a hypothesis, not as authority. Before downloading or modifying anything, verify the current facts against: 1. The official Qwen3.8 model card: https://huggingface.co/Qwen/Qwen3.8-27B 2. The selected GGUF repository: https://huggingface.co/bartowski/Qwen3.8-27B-GGUF 3. Current llama.cpp documentation and releases: https://github.com/ggml-org/llama.cpp 4. Current Hermes documentation: https://github.com/NousResearch/hermes-agent 5. Directly relevant first-hand issue reports for Qwen3.8, llama.cpp, Hermes, Windows, CUDA, and RTX 3090 hardware. Check whether any filenames, flags, templates, runtime requirements, compatibility guidance, or known defects have changed. Record every material correction to this plan, including: - The original assumption - The contradicting evidence - The corrected decision - The supporting source Do not execute a command merely because this prompt supplied it. Investigate the first-hand report that a similar model used approximately 18–20 GB plus 10–12 GB of context-related memory, spilled into system RAM, and generated only 4–6 tokens per second on two GTX 1080 Ti GPUs with 11 GB VRAM each. Determine whether that experience applies to this single RTX 3090 by accounting for: - Exact model and quantization - KV-cache type - Context length - Hybrid attention architecture - GPU memory bandwidth and generation - Single-GPU versus multi-GPU execution - PCIe and cross-GPU overhead - CPU or system-RAM offloading - Number of parallel inference slots - Vision-projector memory - MTP/speculative-decoding memory Do not use the friend’s performance result as either a guarantee or dismissal. Use it as a hypothesis to test. TARGET ARTIFACTS The provisional selection is: Repository: bartowski/Qwen3.8-27B-GGUF Primary model: Qwen3.8-27B-Q4_K_M.gguf Vision projector: mmproj-Qwen3.8-27B-bf16.gguf Canonical directory: C:\AI\Models\Qwen3.8-27B\ Before downloading, verify that these remain compatible and appropriate. If current evidence identifies a materially better artifact for this exact hardware and agent use case, stop and explain the evidence before changing the selection. Do not silently substitute: - A generic Ollama tag - A different quantization - An unofficial fine-tune - A text-only model - A model without the compatible vision projector PREINSTALLATION AUDIT Inspect and record: - Windows version - Exact RTX 3090 identity and current VRAM use - NVIDIA driver version - Whether CUDA is visible to the selected runtime - CPU model and available system RAM - Free disk space - Existing Qwen files - Existing Ollama, LM Studio, llama.cpp, or model-server processes - Occupied local ports - Existing Hermes installation and configuration - Existing local/cloud model providers - Whether an identical verified GGUF already exists Require sufficient free disk space for: - Model weights - Vision projector - llama.cpp binaries - Logs and temporary download files - A future comparison quant if authorized Do not overwrite or delete existing model files or Hermes settings. DOWNLOAD Use a resumable Hugging Face download. Download the exact model and projector into the canonical directory. Verify after download: - Exact filenames - Expected approximate sizes - Completed, non-partial files - Repository revision - Available SHA-256, Xet, or Hugging Face integrity metadata - Local file hashes recorded in the installation receipt If interrupted, resume rather than restarting from zero. LLAMA.CPP RUNTIME Use a current Windows CUDA-enabled llama.cpp release known to support Qwen3.8’s architecture correctly. Do not assume the Winget package contains the required CUDA backend. Prefer an official prebuilt CUDA Windows release unless source compilation is necessary. If compiling, use the RTX 3090’s appropriate CUDA architecture and current documented build flags. Keep the executable and all llama.cpp/CUDA DLLs from the same release together. Do not mix: - A new llama-server executable with old shared libraries - CUDA and non-CUDA builds - Libraries from separate llama.cpp releases Verify before loading the model: - llama-server version and commit/build number - CUDA backend availability - RTX 3090 appears in the runtime’s device list - No stale llama.cpp libraries are loaded from another directory Use a build at least as recent as the known Qwen3.8 CUDA fixes, but verify the current known-good release rather than blindly trusting a hard-coded build number. NETWORK BOUNDARY Bind the model server only to: 127.0.0.1 Do not bind to: 0.0.0.0 Do not expose it to the LAN or internet. Use port 8080 unless it is already occupied. BASELINE TEST PHASE Start with: - Q4_K_M model - All feasible model layers on the RTX 3090 - One inference slot - Flash Attention enabled - Q8_0 K cache - Q8_0 V cache - MTP disabled - Parallel tool calls disabled - Correct embedded or explicit Qwen Jinja chat template - Context length: 32,768 - Reasoning effort: moderate for the baseline - Prompt caching enabled only if currently recommended and stable For the first text-only test, do not load the vision projector. Confirm from logs: - The RTX 3090 is actually used - Expected model layers are GPU-resident - No accidental CPU-only execution - No unexpected system-RAM spill - Model output is coherent - Chat formatting is correct - Thinking blocks and final answers are parsed correctly - No corrupted characters or malformed output Record: - Model-load time - Prompt-processing speed - Generation speed - Idle and peak VRAM - System RAM use - CPU use - Exact server command - Exact runtime version VISION TEST Restart with the compatible BF16 vision projector. Repeat the memory measurements. Test at least: - One ordinary image description - One screenshot containing interface elements - One screenshot requiring a specific visual observation Confirm the image reaches the model and is not silently ignored. CONTEXT LADDER Do not immediately configure the advertised maximum context. Test sequentially: 1. 32K — safe baseline 2. 48K — likely everyday compromise 3. 64K — desired Hermes target At each level: - Restart cleanly - Run the same representative prompt - Record reserved and peak VRAM - Record system RAM - Record prompt-processing and generation speed - Check for CPU or RAM spill - Check for instability or malformed output - Run a multi-turn tool-use conversation Use one inference slot for all initial comparisons. The server context and Hermes configured context must match. Do not configure Hermes to believe it has more context than the server actually provides. Selection rule: - Keep 32K if higher settings materially destabilize the system. - Prefer 48K if it provides reliable agent operation with comfortable VRAM headroom. - Use 64K only if repeated workflows remain stable and measured peak VRAM stays within a safe limit. - Treat 128K or greater as experimental. - Do not reduce the model quantization merely to advertise a larger context without proving that the larger context improves real task success. If 64K is slightly too large: - First inspect exact KV and buffer allocation. - Consider context reduction. - Consider supported KV placement or cache adjustments. - Do not use Q4 KV caching without measuring prompt-processing performance. - Do not silently spill major model layers into system RAM. HERMES CONFIGURATION Preserve the existing Hermes configuration. Create a separate local-model profile or equivalent isolated configuration. Configure: Provider: custom Base URL: http://127.0.0.1:8080/v1 Model: the exact llama.cpp model alias Context: the verified selected context length Confirm that Hermes status displays: - Custom/local provider - Correct model name - Correct local endpoint - Correct context length Prevent unintended cloud-model fallback. Do not delete cloud credentials or other profiles. Isolate the local profile instead. Distinguish: - Local model inference - External web or browser tools A local model may still use external tools if those toolsets are enabled. Do not describe the whole session as offline merely because inference is local. For the local-only verification profile: - Disable cloud inference fallback - Disable paid managed tool fallback - Keep only intentionally selected local tools - Monitor llama.cpp request logs - Confirm no model request reaches Nous, OpenRouter, OpenAI, Anthropic, or another remote inference provider CHAT TEMPLATE AND TOOL-CALL VALIDATION Verify that the server uses the correct Qwen chat template. If Hermes outputs: - Tool calls as ordinary JSON text - XML-like tool syntax instead of structured calls - Corrupted output - Repeated malformed assistant turns - Incorrect role boundaries stop and inspect: - Embedded Jinja template - Official chat_template.jinja - llama.cpp tool-call parser - Hermes OpenAI-compatible request format - Parallel-tool configuration Do not blame the quantization until the template and runtime are ruled out. Run these workflows three times each: 1. Ask Hermes to run harmless system-information commands and report the actual results. 2. Ask Hermes to create a temporary file with exact known content. 3. Ask Hermes to read the file back and compare it byte-for-byte. 4. Ask Hermes to modify the file using information returned by the previous tool. 5. Ask Hermes to recover from one intentionally invalid path. 6. Ask Hermes to inspect a screenshot and perform a harmless computer-use action in a test environment. For dependent, state-changing actions, require sequential tool calls until parallel behavior has been separately validated. Reject any run where Hermes invents the result of a tool it did not successfully execute. MTP PHASE Do not enable MTP speculative decoding until: - Text output is clean - Vision works - Structured tool calls work - Context is stable - Three repeated Hermes workflows pass Then test MTP separately using the currently documented Qwen3.8 llama.cpp flags. Compare with and without MTP: - Generation speed - Prompt-processing speed - VRAM - Stability - Output correctness - Tool-call correctness - Long-context behavior - Repeated and concurrent requests Keep MTP only if it provides a material measured improvement without: - OOM errors - CUDA lockups - Corrupted tokens - State leakage - Tool-call regressions AUTONOMY CONFIGURATION Only after the verified Hermes tests pass: - Enable the required terminal, file, browser, computer-use, skill, memory, and delegation toolsets. - Configure Hermes’s supported approval-off/high-autonomy mode. - Retain checkpoints or backups. - Keep the Windows account non-administrator unless a specific task genuinely requires elevation. - Preserve Hermes’s irreducible catastrophic-command protection. - Do not expose secrets or production credentials to the local agent by default. Verify the autonomy setting with harmless commands before unattended work. PI AND OTHER HARNESSES Do not download another model copy for Pi. Point Pi or other compatible harnesses at the same local endpoint where possible. If a harness cannot use the OpenAI-compatible endpoint and requires direct GGUF access, point it to the canonical model file. Do not allow a harness to silently import or duplicate the weights without reporting where the copy is stored. QUALITY AND PERFORMANCE EVALUATION Do not declare Q4_K_M universally best from one chat response. Create a repeatable evaluation set covering: - Repository understanding - Coding and debugging - Structured tool calls - Terminal operations - File editing - Browser operation - Computer use - Recovery from failed actions - Instruction following - Long-context retrieval - Hallucination resistance Record: - Successful completed-task rate - Failure categories - Median and worst-case latency - Tokens per second - Prompt-processing speed - VRAM and system RAM - Manual intervention count - Incorrect or fabricated tool results Use this baseline later to compare Q5_K_M or another model. Do not download comparison models until the Q4_K_M baseline is complete and the additional download is explicitly approved. ROLLBACK AND DOCUMENTATION Before modifying Hermes: - Back up its existing configuration. - Record all changed values. - Preserve existing provider credentials without printing them. Produce: - Installation receipt - Exact model and projector hashes - Source repository and revision - llama.cpp build/version - Final launch command - Final selected context - Hermes local-profile configuration - Local-only verification evidence - Performance measurements - Known limitations - Startup and shutdown instructions - Rollback procedure - Removal instructions that do not delete unrelated models or user data Do not create an automatic Windows startup service until the configuration passes verification and the user approves persistent startup. STOP GATE Do not call the setup complete merely because the files downloaded or the server started. Completion requires: - Correct model and projector verified - CUDA-enabled llama.cpp verified - RTX 3090 actually used - Coherent text output - Vision input verified - Structured tool calls verified - No unintended cloud-model calls - Selected context verified through measurement - Three repeated Hermes workflow passes - No fabricated tool results - Stable memory behavior - Exact configuration and rollback documented If any required criterion fails, report: - The exact failed criterion - Evidence - Likely cause - Smallest safe next correction Do not conceal failures, silently substitute another model, or describe an incomplete setup as production-ready.