A partner asks whether the firm should buy its own AI system so client files stay private. This post gives a straight answer on private AI for law firms and accounting firms. For most small practices, the answer is no, you do not need to buy a server. What you need is to know where client data goes when someone pastes it into a tool, and whether that place fits your confidentiality duties. Below, we rank the options by where the data goes, quote the rules worth reading first, and say when private AI is the wrong project.
Most of the decision comes down to four questions: what gets sent, where it is stored, who can read it, and whether it is used to train a model. Answer those for each tool your staff use, and the right option usually becomes clear. Read on for the options, the rules, and a first project that keeps the risk small.
What “private AI” means
There are two private options. Open-weight models run on hardware you own. Commercial models run inside your own Azure or AWS account, under terms that bar training on your data. Either way, the model runs in your own environment or your own cloud account. We size both options against your data, budget and accuracy needs, and tell you which one fits.
If your staff are already pasting files into chatbots, read the shadow AI post before you decide anything else.
Three options, ranked by where client data goes
Each option sends client data to a different place. Read the table from top to bottom, least control to most. The ranking is about where data goes, not about which vendor is best. Each row can be the right answer for a different firm, and the wrong answer for another.
| Option | Where the data goes | Who it fits |
|---|---|---|
| Consumer chatbot on a personal account | The vendor’s servers, under the vendor’s consumer terms | Public information only, such as general research on published rules. Not client files |
| Licensed tool inside your existing Microsoft 365 tenant, for example Microsoft 365 Copilot | Inside your Microsoft 365 tenant, with the permissions you already have configured | Firms already on Microsoft 365 that have reviewed who can see which files |
| Private AI on hardware you own, or in your own Azure or AWS account | Your hardware, or your cloud account, under terms that bar training on your data | Firms with a large or sensitive document set and someone to own the system |
The Microsoft 365 option needs a permissions review before rollout. Copilot inherits your permissions, including the bad ones. Our Microsoft 365 Copilot vs Private AI guide covers how to choose between the two paths.
The consumer option is the easiest one for staff to start using on their own. A personal account is easy to open and gives the firm no view of what was pasted into it. That gap, not the quality of the answers, is the main reason to set a policy and offer an approved tool.
What the professional rules say
For tax preparers and accounting firms. The FTC Safeguards Rule (16 CFR Part 314) applies to financial institutions, and its examples include “an accountant or other tax preparation service that is in the business of completing income tax returns.” The rule includes a service provider oversight requirement. Section 314.4(f)(1) calls for “taking reasonable steps to select and retain service providers that are capable of maintaining appropriate safeguards for the customer information at issue.” The rule defines a service provider as any person or entity that “receives, maintains, processes, or otherwise is permitted access to customer information” through the services it provides to the firm. Read that definition against any AI vendor that will hold client files, and document how you checked the vendor. In practice, that means putting the questions in the last section to the vendor, getting the answers in writing, and keeping that record in the firm’s files, not in one employee’s inbox.
For lawyers. The Illinois Supreme Court announced its policy on AI in the courts on December 18, 2024. The announcement says “The Rules of Professional Conduct and the Code of Judicial Conduct apply fully to the use of AI technologies.” It also says that before using any technology, including generative AI applications, “users must understand both general AI capabilities and the specific tools being utilized.” The Illinois Rules of Professional Conduct include Rule 1.6, Confidentiality of Information. Paragraph (e) of that rule says a lawyer “shall make reasonable efforts to prevent the inadvertent or unauthorized disclosure of, or unauthorized access to, information relating to the representation of a client.” So the first job for a law firm is the same as for an accounting firm: know where client information goes before a tool is approved.
This is not legal advice. Each firm should check its own bar or board rules before it sets an AI policy.
When private AI is the wrong project
Private AI is not the answer for every firm. Skip it if any of these describe your practice:
- Your document volume is small. Small document sets rarely justify a custom build.
- A licensed tool already covers the work. If your Microsoft 365 setup is in order and the tool does the job, use it.
- No one will own the system. A private system has to be secured, documented and kept that way, and someone has to answer when it breaks. If nobody has that time, do not start. Name that person before the project starts, not after the first failure.
- Your files are messy. Documents spread across personal drives, with no folder structure and no owner, produce messy answers. Clean up the source first.
If one of these describes your firm, we will say so on the first call. If there is nothing worth building, we will tell you. A firm that hears “you do not need this” before it spends money has been helped, even if no project follows.
A sensible first use case
Start with document Q&A over one approved document collection, such as your firm’s internal policy manual or a set of engagement templates. Give access to a small pilot group. Check the answers against the source documents, and expand only after the pilot works on real questions. A pilot like this fits a fixed-scope project with a defined deliverable.
Good pilot questions are the ones staff already answer by searching folders, such as which version of a form is current or what the steps are for a new intake. Record every wrong answer. A wrong answer is a finding about the document set or the setup, and it is the reason to run a pilot in the first place.
Questions to ask any vendor (ours included)
Ask these before you sign anything, whether the vendor is a software company, an integrator or us. A vendor that answers each one plainly and in writing is easier to work with. A vendor that cannot answer them is telling you something about the project.
- Where is our data stored, and in which country and region?
- Who can access it, including your staff and any subcontractors?
- Is our data used to train models? Get the answer in writing.
- What is logged, who can read the logs, and how long are the logs kept?
- Can we export and delete our uploads, prompts and outputs?
- If we leave, what do we get back, in what format, and how is our data deleted?
- Who owns the configuration, prompts and workflows you build for us?
Questions 2 and 4 come down to access controls and audit logging. Our cybersecurity services page covers that work.
Next step
Start with an AI integration assessment. Bring the list of tools your staff already use, even if you only know half of it. That list is the fastest way to see where client data is going today. In the assessment, we map where your team’s hours go and where your data lives, then tell you which option fits your firm. Details are on the AI Integrations page. To talk it through, book a 30-minute call or call 224-531-1000. The call is a chance to talk through the work that eats your team’s week.
