Classifying Content with Your Own AI Questions
Some questions about your content can’t be answered with a pattern or a rule. A rule can find a credit card number, but it can’t tell you whether a document is a contract, when that contract ends, or whether an email is really about someone’s salary. Records, legal and HR teams usually answer questions like these by opening files one at a time.
With AI analysis, Shinydocs Pro asks a language model your own questions about each document and saves the answers on the document as fields. You can then filter on those answers in the Catalog, combine them with PII and ROT results, report on them and tag them, across every source you choose.
This guide works through two examples from start to finish:
- Find contracts, what kind they are, when they end and whether they renew automatically.
- Flag documents that discuss the pay of specific people.
For a description of every setting on the AI page, see AI Enrichment (Content Tagging).
How AI analysis works
- You set up each question as a prompt under Settings > Analysis > AI. A prompt says which documents to read, which model answers, what to ask and which answers to keep.
- When a source runs AI analysis, Shinydocs Pro sends each matching document to the model with your instructions. The model fills in the answer fields you defined, and each answer is saved on the document as
ai-text-analysis-<field name>. - AI analysis runs after every other analysis on the source has finished.
- Each document is answered once. After that, each update sends only documents that haven’t been answered yet, such as new and changed files. Changing a prompt doesn’t change the answers documents already have; you choose when to ask again.
Before you start
- The Intelligence Engine add-on. AI analysis is part of the Intelligence Engine add-on. Check that it’s listed under Add-ons on Settings > License. Trial licenses don’t include it. To add it, contact Shinydocs.
- A model that generates text. Under Settings > Intelligence, add a provider and switch on at least one model that generates text. See the Intelligence Configuration Guide (for Search and AI Enrichment), or Shinydocs AI Interface Quick Start to start from scratch. AI analysis asks the model to answer in a fixed format, and not every model can: the GPT-OSS models, for example, can’t be used.
- The AI Analysis Tool feature flag. Open your Control Center address followed by
/settings/flags, for examplehttps://localhost:9701/settings/flags, and set AI Analysis Tool to Enabled. Until it’s on, the AI tab isn’t shown and AI analysis doesn’t run. - Text to read. AI analysis reads the text Shinydocs Pro extracted from each file. Turning it on for a source also turns on Reading file contents. A file with no text, such as a scan with no readable text, gets no answer.
- A test folder. Pick a folder you know well that holds a few dozen documents: some that should match your question and some that shouldn’t. You’ll try your prompts there first.
Before you change a feature flag. Feature flags are on their own page, not in the Settings menu, and a change saves straight away. A flag applies to every user, and some flags can change or remove data. Turn on only the flags a Shinydocs guide or Shinydocs Support tells you to, and leave the rest at Default. Don’t select Reset all: it also switches off flags you turned on.
Decide where your documents go. A prompt sends what its placeholders bring in to the provider of the model you choose. With the starting user prompt, that’s each document’s name, location and text. With a provider that runs on your own hardware, your content stays on your network.With a cloud provider, it leaves your environment. For contracts, pay records and other sensitive content, use a model that runs on your own hardware. See Shinydocs Private AI.
Answers are saved in the index like any other metadata, so they show in the Catalog and in reports. Don’t ask the model to copy sensitive details, such as names or amounts, into an answer.
Example 1: find contracts and their renewal terms
This example uses two prompts. The first reads the start of every document and decides whether it’s a contract. The second reads the whole of each contract and finds when it ends and whether it renews automatically. Splitting the work this way keeps the long reads for the documents that need them.
Prompt 1: Contract screen
- Go to Settings > Analysis and open the AI tab. Select Add prompt and enter the Name
Contract screen. - Under Documents, open Sources, tick the sources that hold contracts and select Apply. A source you add later isn’t included until you tick it here.
- Under Conditions, keep the two conditions every new prompt starts with.
fullTextexists skips files with no text, andpath-validis notfalseskips files that are no longer in the source. Select Add condition twice and set:extensionis one ofpdf,docx,doc. Extensions are in lowercase, without the dot.parentis your test folder, for example\\fs01\shares\Legal\Contracts\AI-test.parentis the folder a file is in; copy the path from theparentcolumn in the Catalog to get it exactly right. The condition also matches files in folders inside it, so choose a small folder. You’ll remove this condition once the prompt works.
- Under Model, choose the Model to use. Only models that are switched on and generate text are listed. For some models, a Thinking setting appears. Leave it at Use the model’s setting unless the answers need more reasoning: more thinking makes every document slower.
- Under Speed, leave Documents at a time empty to use the model’s own setting.
Unknown Attachment
- Under Instructions, enter this System prompt:
You are a records analyst. Decide whether the document is a contract or agreement between two or more parties, including amendments, renewals, statements of work and leases. Drafts and signed copies both count. Policies, invoices, quotes and letters that only mention a contract do not.
is_contract: true if it is a contract, otherwise false.
contract_type: if it is a contract, exactly one of: Service agreement, Software license, Lease, Non-disclosure agreement, Purchase agreement, Employment agreement, Amendment, Other. If it is not a contract, answer N/A.
- Leave the User prompt as it is. It sends the document’s name, location and text, then repeats your system prompt after the text.
- Set Document text limit to
4000. The first 4,000 characters are usually enough to tell a contract from other documents, and sending less makes each document quicker. - Under Answer, fill in the first row of Fields, then select Add field for the second:
| Field name | Type | Required | What it holds |
|---|---|---|---|
is_contract |
Yes or no | Ticked | true for a contract, false for anything else |
contract_type |
Text | Ticked | One of the types listed in the prompt; empty for anything that isn’t a contract |
- Leave Answers to ignore as it is. When a document isn’t a contract, the model answers
N/Afor its type. That answer isn’t saved, so the field stays empty.
Unknown Attachment
Type field names in lowercase, with underscores between words; anything else you type becomes an underscore. Each answer is saved with ai-text-analysis- in front of its name, so is_contract becomes ai-text-analysis-is_contract. A required field is always answered; the model can leave the others empty. See Metadata captured during content analysis for the other fields AI analysis writes.
Prompt 2: Contract terms
- Select Add prompt, name it
Contract termsand choose the same Sources. - Under Conditions, select Add condition and open the new row’s field list. Answer fields from your other prompts come first, each marked with the prompt that answers it. Choose
ai-text-analysis-is_contract, then is, and typetrue.
Unknown Attachment
Because this condition uses an answer from Contract screen, Shinydocs Pro always runs Contract terms after it, on the documents it marked as contracts. While you test, that’s only contracts in your test folder, because only they have an answer.
- Choose the same Model to use. Prompts that use the same model run together, so the model is loaded once.
- Enter this System prompt:
The document is a contract. Answer from its text only.
end_date: the date the contract ends, or the end of its current term, as YYYY-MM-DD. If no end date is stated, answer N/A.
auto_renews: true if the contract renews or extends automatically unless someone gives notice, otherwise false.
Do not include the names of people or organizations in any answer.
- Leave Document text limit at
0so the whole contract is read. End dates and renewal clauses are often near the end. - Add these Fields:
| Field name | Type | Required | What it holds |
|---|---|---|---|
end_date |
Text | Not ticked | The end of the contract or its current term, such as 2027-03-31; empty when none is stated |
auto_renews |
Yes or no | Ticked | true when the contract renews unless someone gives notice |
Ask for every date in the same form,YYYY-MM-DD, and forN/Awhen there isn’t one. Answers written the same way filter consistently in the Catalog, and anN/Ais left out instead of being saved.
Example 2: flag documents that discuss individual pay
PII rules find identifiers such as social insurance numbers, but not an email about a named employee’s raise or a spreadsheet of bonuses. This prompt finds documents like these wherever they’re kept, so HR and privacy teams can decide what belongs in the HR system and what should be removed.
- Select Add prompt, name it
Pay discussionsand choose the Sources to check, such as shared drives and mailboxes. - Under Conditions, keep the two starting conditions. Add
extensionis one ofdocx,doc,pdf,xlsx,xls,msg,eml, and the sameparentis condition for your test folder. - Choose the Model to use and enter this System prompt:
You review documents for an HR privacy check. Decide whether the document states or discusses the pay of specific, identifiable people, such as a named employee's salary, raise, bonus, commission or severance, or a list of employees with their pay. Salary bands, pay scales and job postings that don't name anyone do not count.
discusses_pay: true or false.
pay_topics: the kinds of pay discussed, from this list only: salary, raise, bonus, commission, severance, benefits. An empty list if none.
pay_reason: one short sentence explaining your answer, without names, employee numbers or amounts.
- Add these Fields:
| Field name | Type | Required | What it holds |
|---|---|---|---|
discusses_pay |
Yes or no | Ticked | true when the document discusses a specific person’s pay |
pay_topics |
List of text | Not ticked | The kinds of pay discussed, such as salary and bonus |
pay_reason |
Text | Not ticked | A short reason for the answer, to help you check it |
Unknown Attachment
The short reason makes answers quicker to check. Asking for it without names or amounts keeps the answer from copying sensitive details into the index. A model doesn’t always follow that instruction, so check the reasons in your sample too.
Set up all the prompts for a source before its full run. Shinydocs Pro marks a document as done once every prompt that applies to it has answered. A prompt you add later reaches documents that are already done only when they change, or when you choose Re-run, which asks every prompt again on every document they match.
Test on a small set first
Once a document has an answer, it isn’t sent again unless it changes or you ask for it. Getting a prompt right on a few dozen documents takes minutes. Fixing it after a whole source has run means running the whole source again.
- Select Save changes at the bottom of the AI tab. A prompt can’t be saved until it uses at least one document placeholder, such as
; the starting user prompt already does. - Go to Sources and open the source. In its Analysis panel, select AI analysis to open its card, then select Switch on and start (or Start, if its switch is already on). This runs AI analysis now and keeps it on for the source’s updates.
Unknown Attachment
To turn AI analysis on for several sources at once, go to Settings > Analysis and stay on the General tab. Choose Default, a connector or a source, turn on AI analysis and select Review & save. It then runs with each source’s next update, or straight away with Run now on the source page.
- When the card shows the run has finished, open the Catalog. Select Columns, the icon at the right end of the column headings, type
ai-text-analysisand tick the answer fields. - To see only the documents that were answered, open the menu on the
ai-text-analysis-is_contractcolumn heading and choose Filter by condition > Is not empty. - Check the answers against what you know about the test documents. Look at the
falseanswers as well as thetrueones: a contract the model missed is easier to overlook than one it got wrong. - If answers are wrong, change the prompt and select Save changes. Common fixes are to say what doesn’t count, to list the exact answers you want, or to choose a larger model. Then, on the source’s AI analysis card, select the arrow next to Run again and choose Re-run. It runs your prompts again on every file they match, which, while the test condition is in place, is only the test folder.
- Repeat until you’re happy with the answers.
Run it on all your documents
- On the AI tab, open each prompt and remove the
parentcondition with Remove condition, the × at the end of its row. Select Save changes. - On the source page, select Run again on the AI analysis card, or wait for the next update. Documents in the test folder keep their answers, and the rest of the source is sent.
- Follow progress on the card. After the full run, each update sends only documents that haven’t been answered yet, such as new and changed files.
Keep runs manageable
Every document AI analysis sends waits for the model to answer, so a large source can take a long time. These make a run longer:
- More documents. Use sources and conditions to send only the documents that need asking.
- More prompts. Each prompt is a separate request for every document it applies to.
- More text. Lower the Document text limit when the start of a document is enough. If long documents fail, lowering it helps too: a model reads only as much as its context window holds.
- Thinking and larger models. They can give better answers to hard questions, but every document takes longer.
Documents at a time works the other way: sending more documents at once finishes sooner when the provider has capacity, but puts more load on it. Some servers, such as LM Studio, share the model’s context window between the documents they’re sent. With a cloud provider, every document sent is a request on your account with that provider.
If the provider keeps failing, refuses the run or can’t load the model, the run stops and the card says so. Files that weren’t done are picked up by the next run.
Use the answers in the Catalog
Answer fields work like any other field in the Catalog. Add them with Columns, then use the menu on each column heading to filter. Filters on different columns combine.
| Answer type | How to filter it |
|---|---|
| Yes or no | Filter by values > Is any of…, then tick true or false |
| Text | Filter by values > Is any of… and type the values, or Filter by condition > Contains… or Starts with… |
| List of text | Filter by values > Is any of… or Is all of… and type the values |
| Any type | Filter by condition > Is empty or Is not empty |
Some examples:
- Contracts that renew automatically:
ai-text-analysis-is_contractistrueandai-text-analysis-auto_renewsistrue. Add theai-text-analysis-end_datecolumn to see when each term ends. - Contracts that end in a given period: open the menu on the
ai-text-analysis-end_datecolumn and choose Filter by condition. If the answers were saved as dates, it offers Is before…, Is after… and Between…. If they were saved as text, choose Starts with… and type, for example,2027-. - One kind of contract: on
ai-text-analysis-contract_type, choose Filter by values > Is any of… and typeLease. - Pay details next to identifiers:
ai-text-analysis-discusses_payistrue, and on the PII column, Filter by condition > Is not empty. These files discuss someone’s pay and also hold personal identifiers, so review them first. See Managing Sensitive Content to Reduce Reputational Risk. - Copies you don’t need: combine an answer with the ROT or Duplicates column, for example to find contracts that are also flagged as redundant.
Select Save filters to keep a set of filters, and open it again from Saved filters. For more about the Catalog, see Getting to Know Content Catalog & Creating Reports in Shinydocs Pro.
Report, tag and act
With the Catalog filtered, select Action. With no files ticked, an action covers every file that matches your filters.
- Report lists the files in a CSV file. Export all fields is ticked to start with; untick it to export only the columns you’re showing, such as the answer fields. Download the report from View activity > Remediation.
- Tag records what a person decided, for example a tag
Contract-reviewwith the valueConfirmedon the contracts someone has checked. Tagging needs the Tagging in Catalog feature flag. See Tagging items in the Shinydocs Pro Catalog. - Dispose deletes the files from their original location. Use it only on files a person has reviewed. Only Control Center administrators can dispose of files, and a disposal can’t be undone. See Disposal in Shinydocs Pro.
Check the quality of the answers
A model’s answers are suggestions, not decisions. How good they are depends on the model, your prompt and your documents, so measure them on your own content.
- Spot-check a sample. After each full run, open a handful of documents for each answer: some
true, somefalse, and some of each contract type. If the same kind of mistake keeps appearing, change the prompt. - Look for documents with no answer. Filter the answer column with Is empty, together with the prompt’s own conditions, such as the Extension column. Add the
ai-text-analysis-errorcolumn to see why a document wasn’t answered, for exampleNo text to analyze. - See where AI analysis has finished. Add the Analysis column. It lists the analyses each file has been through, and shows AI analysis once AI analysis has finished with the file, including a file it couldn’t answer.
- See which model answered. Each answer has a field beside it that names the model, such as
ai-text-analysis-is_contract-model-used. It helps when you compare models. - Ask again for a few documents. Tick them in the Catalog, select Action > Tag > Remove tag and choose the answer field; AI analysis fields are shown in green. The answer is removed, and the next AI analysis run answers the document again with your current prompts. For many documents, use Re-run on the source page.
- Keep a person in the loop. Before you dispose of files or report them as findings, have someone confirm the answers. Tag what they confirmed and act only on the confirmed files.
What AI analysis can’t do
- It reads only the text Shinydocs Pro extracted. Images and scans with no readable text get no answer.
- It reads only as much of a document as the Document text limit and the model’s context window allow. A clause past that point isn’t seen.
- It judges each document on its own. It can’t compare documents with each other.
- It doesn’t ask again by itself. Editing a prompt, adding a prompt or changing the model leaves existing answers as they are until the document changes or you choose Re-run.
- Renaming or removing an answer field doesn’t remove the answers already saved under the old name. Remove them in the Catalog with Remove tag.
- Its answers can be wrong, and a model doesn’t always follow every instruction.
Let’s review
By using AI analysis in this way, you can:
- Ask your own questions about each document and keep the answers as fields you can filter, report on and tag in the Catalog.
- Send only the documents that need asking, and chain prompts so the long reads happen only where they’re needed.
- Get each prompt right on a small test folder before you run it on a whole source.
- Combine the answers with PII, ROT and other columns to find what matters most.
- Keep sensitive content on your own network, and keep a person’s review before any action that can’t be undone.



