Skip to content
English
  • There are no suggestions because the search field is empty.

CLI: Migration: File Share to SharePoint Online

 CLI: Migration: File Share to SharePoint Online

Software: Cognitive Toolkit (CognitiveToolkit.exe)

Operation: Migrate

Use the Cognitive Toolkit’s Migrate operation to copy files that Shinydocs Pro has crawled from a file share into a SharePoint Online document library, keeping their folder structure.

  • File contents are copied, and each file gets the created and modified dates of the source file.
  • Permissions aren’t carried over. The files get the library’s permissions.
  • Columns can be filled from index fields with a mapping file.
  • The source files stay where they are.
Before you start
  • The Cognitive Toolkit is in the Control Center folder, by default C:\Program Files\Shinydocs Professional\ControlCenter. Activate it once with your Shinydocs Pro license file, as the Windows account that will run the migration: CognitiveToolkit.exe activate -p "C:\Shinydocs\license.xml". Keep the license file where it is: the Cognitive Toolkit reads it from that path each time. Your license must include migration.
  • The source must already be crawled by Shinydocs Pro. To find its index name, open the source in Control Center and look under Index information: you can use its Name or its Alias, and each has a copy button.
  • Register an application in Microsoft Entra ID that signs in with a certificate, and grant it SharePoint application permissions that let it add files to the destination site, with admin consent. Microsoft’s guide uses Sites.FullControl.All; to limit the app to chosen sites, use Sites.Selected and give it write access to the destination site. See Microsoft’s Granting access via Entra ID App-Only. SharePoint Online only accepts certificates for app-only access.
  • Put the certificate’s .pfx file on the machine that runs Migrate, and note the app’s application (client) ID and your directory (tenant) ID.
  • The account that runs Migrate needs to read the source files.
  • Give every file as a full path.
Steps

1. Create the destination settings file

Save this as, for example, C:\Shinydocs\sources\sharepoint-online.json:

JSON

{   "sourceType": "sharepoint-online",   "sourceConnectionSettings": {     "site-url": "https://contoso.sharepoint.com/sites/Records",     "app-id": "00000000-0000-0000-0000-000000000000",     "tenant-id": "00000000-0000-0000-0000-000000000000",     "cert-path": "C:\\Shinydocs\\Certificates\\contoso-migrate.pfx",     "cert-password": "~spo-cert-password"   } } 
Key Value
site-url The URL of the site to migrate into.
app-id The application (client) ID of the app registration.
tenant-id Your directory (tenant) ID.
cert-path Full path to the certificate’s .pfx file. In JSON, write each backslash twice.
cert-password The .pfx file’s password, if it has one. Use a tilde (~) followed by a value you saved with CognitiveToolkit.exe savevalue; see CLI: Encrypting repository passwords with SaveValue. Leave the key out if the file has no password.

The key names are case-sensitive, and sourceType must be sharepoint-online.

2. Create the query file

The query file holds only the query, with no "query" wrapper. To migrate everything in the index, save this as C:\Shinydocs\queries\match_all.json:

JSON

{ "match_all": {} } 

For more examples, see CLI: Common Query Types.

3. Run Migrate

Open Command Prompt as the account that will run the migration, go to the Control Center folder and run Migrate. For example:

CognitiveToolkit.exe Migrate --index-server-url "http://localhost:9200" --index-name "finance-share" --query "C:\Shinydocs\queries\match_all.json" --destination-source-settings "C:\Shinydocs\sources\sharepoint-online.json" --path-prefix-to-remove "\\FS01\Finance" --start-location "Shared Documents" 

--start-location is the library’s address within the site, not its display name. For the default Documents library, that’s Shared Documents. With these values, \\FS01\Finance\Budgets\2026\plan.xlsx goes to the Budgets/2026 folder of that library.

To check your settings first, add --dry-run: Migrate goes through the items without uploading anything or updating the index.

Check the results

Migrate shows its progress as it goes, and ends with an exit code:

Exit code Meaning
0 Every item the query found was migrated.
1 Some items failed, or a required parameter was missing.
5 A license problem, for example a license without migration.
-1 Migrate stopped, for example because a settings file isn’t valid JSON or sign-in to the destination failed.

Details of each failure are in the Cognitive Toolkit’s log file, CognitiveToolkit<date>.log, in its logs folder.

Migrate also records the result on each item in the index, so you can find failures with the Catalog’s advanced search, for example last-migration-status:error:

Field Value
last-migration-status success or error
last-migration-date When the item was last migrated.
error-string Why the item failed. Empty when it succeeded.
prop-content-id Where the item went: the file’s server-relative URL in SharePoint. Set only when it succeeded.

Running Migrate again processes every item the query finds, so files already uploaded are uploaded again and replaced. To migrate only the items that failed or haven’t been migrated yet, use this query file:

JSON

{   "bool": {     "must_not": [       { "match": { "last-migration-status": "success" } }     ]   } }  File and folder names

Migrate changes names that SharePoint Online doesn’t allow:

  • Each of " * : < > ? / \ | and the text _vti_ is replaced with a hyphen (-).
  • A hyphen is added in front of a name that SharePoint reserves, such as CON, PRN, AUX, NUL, COM1, LPT1, .lock or desktop.ini.
  • Repeated hyphens are then reduced to one, including hyphens that were already in the name.

See also CLI Migration: Characters to Avoid.

Files that already exist

By default, Migrate replaces a file with the same name in the destination folder. To keep existing files, add --disable-over-write: those items aren’t uploaded and are reported as errors.

Filling columns

To fill library columns from index fields, give Migrate a mapping file with --metadata-mapping. Each key is a column’s display name, matched exactly, including capitals, and each value is an index field name:

JSON

[   {     "values": {       "Department": "department",       "Document Type": "doc-type"     }   } ]  Command parameters
Parameter Required Description
--index-server-url, -u Yes The Shinydocs Pro search engine URL, for example http://localhost:9200.
--index-name, -i Yes The source’s index name or alias.
--query, -q Yes Full path to the query file.
--destination-source-settings Yes Full path to the destination settings file.
--start-location Yes The library’s address within the site, for example Shared Documents.
--path-prefix-to-remove No The part of each source path to leave out, usually the folder that was crawled. Without it, only the drive letter or server name is removed.
--disable-over-write No Don’t replace files that already exist. Give it with no value.
--metadata-mapping No Full path to the column mapping file.
--name-field-name No An index field that holds each item’s file name, used instead of its current name.
--threads, -t No How many items to migrate at once. The default is 1.
--dry-run No Go through the items without uploading anything or updating the index.

--location-field isn’t supported for SharePoint Online.