Practical Guide to Privatized Deployment of Domestic Large Models: DeepSeek/Qwen Selection and Cost Calculation

Why do companies need privatized deployment of large models?

As generative AI penetrates into core businesses, enterprises have increasingly stringent requirements for data security and model controllability. Although public cloud services are quick to use, there are three major hidden dangers: there is a risk of leakage when core data is uploaded to a third party, model versions and capabilities are uncontrollable, and long-term usage costs are unpredictable. The privatization deployment of localized large-scale models has become a mainstream choice in industries such as finance, government affairs, medical care, and high-end manufacturing.

Domestic large models represented by DeepSeek and Qwen are outstanding in Chinese understanding and industry adaptability, and are completely open source, supporting enterprises to run independently on their own servers. With the help of privatized deployment, local enterprises in Guangzhou can not only meet the requirements for classification protection and data security compliance, but also control the direction of model iteration in the long term.

Comparison of mainstream domestic large model selections

Private deployment hardware configuration list

Model reasoning has clear thresholds for video memory and computing power. The following configuration is based on the INT8 quantized reasoning requirements:

• Lightweight scenario (Qwen2-7B/14B): single card RTX 4090 (24G) or A10 (24G), 16-core CPU, 64G memory
• Standard scenario (DeepSeek-V2/Qwen2-72B): dual-card A100 40G or four-card A10, 32-core CPU, 128G memory
• High concurrency scenario (72B+concurrency>50): four-card A100 80G + vLLM inference acceleration, combined with load balancing

In terms of storage, model weight files usually require 50-150G SSD space. It is recommended to use NVMe solid-state drives to ensure loading speed.

Deployment architecture selection: vLLM vs Ollama vs Xinference

vLLM

Suitable for high-concurrency enterprise-level deployment, supports PagedAttention acceleration, has high throughput, and has a mature community. Recommended for online inference services in production environments.

Ollama

The first choice for rapid deployment on a single machine. It can be run with one line of commands. It is suitable for POC verification and small-scale internal use.

Xinference

Supports multi-model parallelism and multi-card scheduling, suitable for complex scenarios that require running multiple models at the same time (such as large models + Embedding + vector retrieval).

Five-step method for privatized deployment

Step one: Confirmation of requirements and compliance

Clear the business scenario (conversation/document/code), data sensitivity level, number of concurrent users, and budget limit. Simultaneously start the classification protection or industry compliance assessment.

Step 2: Hardware selection and procurement

Determine the GPU model and quantity based on the model scale and concurrency, and reserve 30% computing power redundancy. If the short-term budget is limited, you can consider the monthly lease transition of bare metal cloud GPU.

Step 3: Model acquisition and quantification

Download open source weights from HuggingFace or ModelScope, and use AWQ/GPTQ/INT8 to quantify and reduce video memory usage. The 7B model requires only about 8G of video memory after INT8 quantization to run smoothly.

Step 4: Deployment of inference framework and API encapsulation

Deploy vLLM or Ollama, configure OpenAI compatible API interface, access the enterprise’s internal knowledge base (RAG architecture), and complete identity authentication and current limiting settings.

Step 5: Integration testing and continuous operation and maintenance

Connect to actual business systems for end-to-end testing, monitor GPU utilization and response delays, and establish model version management and regular security update mechanisms.

Real implementation cases

Case 1: Intelligent Investment Research Assistant for Financial Institutions

A securities company went private and deployed Qwen2-72B to build an internal research report interpretation and Q&A assistant. All research report data are stored locally, and model answers are based on the internal knowledge base, completely avoiding data leakage. The reading efficiency of research reports increased by 5 times, and researcher satisfaction reached 92%.

Case 2: Internal knowledge Q&A in a manufacturing company

A manufacturing company deploys DeepSeek-V2+ private knowledge base, and employees can query product manuals, process standards, and equipment maintenance guides on the intranet. The accuracy of model answers reached 85%, and IT support tickets dropped by 40%.

FAQ

Which is cheaper, private deployment or public cloud API?

When the average daily call volume exceeds 500,000 tokens, privatization is usually more economical; in low-frequency scenarios (<100,000/day), it is recommended to use API verification first before considering privatization.

Can I use the CPU to run large models without a GPU?

Yes, but the inference speed is extremely slow (about 5-10 seconds/word on the CPU for the 7B model) and is only suitable for very low-frequency offline tasks. A GPU is highly recommended for production environments.

Will the model effect become worse after privatized deployment?

Basic abilities remain unchanged. If the enterprise has vertical domain knowledge, it can further improve the answer quality through RAG search enhancement and LoRA fine-tuning, which is usually better than the general API.

Can domestic models meet complex reasoning needs?

DeepSeek and Qwen have reached international first-tier levels in Chinese logical reasoning, mathematical calculations, and code generation, meeting the needs of most enterprise scenarios.

How long does the deployment cycle take?

It usually takes 2-4 weeks from hardware to launch for the standard 7B/14B model; it takes about 6-8 weeks for the 72B large model including data preparation and integration testing.

{“@context”:”https://schema.org”,”@type”:”FAQPage”,”mainEntity”:[
{“@type”:”Question”,”name”:”Private deployment和公有云APIwhere个更便宜？”,”acceptedAnswer”:{“@type”:”Answer”,”text”:”日均调用量超过50万tokens时，Privatization通常更经济；低频场景（<10万/日）建议先用APItest证，再考虑Privatization。"}},
{"@type":"Question","name":"没有GPU可以用CPU跑大模型吗？","acceptedAnswer":{"@type":"Answer","text":"可以，但推理速度极慢（7B模型在CPU上约5-10秒/字），仅适合极低频离线任务。生产环境强烈建议配备GPU。"}},
{"@type":"Question","name":"Private deployment后模型Effect会变差吗？","acceptedAnswer":{"@type":"Answer","text":"基础AbilityforceNo变。若enterprise有垂直领域知识，可通过RAG检索增强和LoRA微调进一步raise升回答质量，通常优于通用API。"}},
{"@type":"Question","name":"国产模型Ability满足复杂推理需beg吗？","acceptedAnswer":{"@type":"Answer","text":"DeepSeek和Qwen在Chinese逻辑推理、数学计算、代码生成上已达到国际一线水平，满足绝大多数enterprise场景。"}},
{"@type":"Question","name":"部署周期需要多久？","acceptedAnswer":{"@type":"Answer","text":"标准7B/14B模型从硬件到上线通常2-4周；72B大模型含Data preparation和集成测试约6-8周。"}}
]}

Data security and compliance points for large model privatization deployment

The core value of privatized deployment is that data does not leave the intranet, but “not leaving the intranet” does not mean “natural security.” Enterprises still need to implement security from three levels: access control, audit logs, and model illusion management. It is recommended to place the private model in an independent security area, unify authentication and current limiting through the API gateway, and record operation logs for each question and answer to facilitate traceability.

At the compliance level, regulated industries such as finance and government affairs need to simultaneously promote grade guarantee assessment and data classification. The corpus used for model training and retrieval should be desensitized and authorized to avoid directly entering raw data containing personal information or business secrets into the database. After the deployment is completed, it is recommended to conduct regular security penetration tests and red team drills, and continue to strengthen the model’s own prompt word injection and unauthorized behavior protection.

From deployment to value: How to integrate large privatized models with business systems

Deployment itself is not the end. The real ROI comes from the deep integration of the model and the business system. It is recommended that enterprises connect large privatization models to the enterprise’s internal knowledge base, so that the model can retrieve enterprise documents, systems and historical cases when answering, rather than relying solely on general knowledge. For relevant implementation ideas, please refer to our Enterprise AI Knowledge Base System practice.

At the same time, the output of private models often needs to be linked with structured data analysis to move from “ability to answer” to “ability to make decisions.” We combine the case of AI data analysis and intelligent BI system to illustrate: When model reasoning and data dashboard are connected, management can directly query operating data using natural language, and the long-term value of privatized deployment will be truly released.

At the project implementation level, you can refer to the tool chains and authoritative documents of the open source community, such as the inference optimization practices published by OpenAI, as an important reference for your own deployment architecture.

Lightweight startup suggestions: Get started without a budget

If the enterprise has a limited budget, there is no need to purchase high-end GPUs right away. You can first use a quantitative compression and distillation solution to run through the POC verification of the 7B model with a single consumer-grade graphics card (such as RTX 4070, RTX 4060) to verify whether the business process and response experience meet the requirements. After the business model is run through and the ROI estimate is clear, you can then decide whether to upgrade to enterprise-level hardware.

Another lightweight path is to first enable public cloud GPU hourly billing for stress testing, collect real concurrency and latency data, and provide a basis for final procurement. This not only avoids premature investment in fixed assets, but also achieves quantifiable results within three months, helping companies verify the feasibility of privatization at the lowest cost.

{“@context”:”https://schema.org”,”@type”:”HowTo”,”name”:”国产大模型Private deployment五步法”,”description”:”从需beg确认到持续运维的大模型Private deployment标准process。”,”totalTime”:”P45D”,”step”:[
{“@type”:”HowToStep”,”name”:”需beg与合规确认”,”text”:”明确Industry务场景、数据敏感度、并发量、预算，同步start up合规评估。”},
{“@type”:”HowToStep”,”name”:”硬件选型与采购”,”text”:”根据模型规模确定GPU型号与数量，预留30%算force冗余。”},
{“@type”:”HowToStep”,”name”:”模型获取与量化”,”text”:”下载开源权重，使用AWQ/GPTQ/INT8量化降低显存占用。”},
{“@type”:”HowToStep”,”name”:”reasoning framework部署与API封装”,”text”:”部署vLLM/Ollama，配置API接口，接入知识库与鉴权。”},
{“@type”:”HowToStep”,”name”:”集成测试与持续运维”,”text”:”end-to-end测试，监控GPU利用率，建立版本管理与安全更新机制。”}
]}