CLI: Running CrawlFileSystem

CrawlFileSystem is the base operation for data discovery and is generally performed prior to running any other Cognitive Toolkit operation.It crawls the specified path (or multiple paths) for metadata. The metadata is then stored in an index where it can be further mined for insights.

Prerequisites

Cognitive Toolkit is a Command Line tool. Every operation follows the same sequence to run:

1. Open CMD as an Administrator or a Service account that has the appropriate permissions.

2. Change directory cd to navigate to where you extracted the shinydocs-extraction-[Version]-[YYYY-MM-DD].zip file (Cognitive Toolkit root folder).

3. After the CognitiveToolkit.exe prompt, type in the runscript command (CrawlFileSystem), followed by the required parameters.

Running the command

To run the CrawlFileSystem command you must provide, at a minimum, the required parameters identified in the following chart:

CrawlFileSystem Options

Example

The following runscript command is an example of the input required for crawling a multiple paths in a file system:

CognitiveToolkit.exe CrawlFileSystem --path-file "
      
       " --index-server-url 
       
         --index-name 
        
          --add-field-owner --after-date-last-modified now-l/d 
        
       
      

Notes

--path-file option

Cognitive Suite 2.6 and later eliminates the need to run the CrawlFileSystem operation more than once by allowing you to specify multiple paths to crawl. A path input file allows you to specify multiple paths to crawl.

Example of a path input file (.txt file):

\\Server2.companyABC.local\Files\HR\Offer Letters \\Server2.companyABC.local\Files\HR\Acceptance Letters \\Server2.companyABC.local\Files\HR\Onboarding \\Server2.companyABC.local\Files\Finance\Purchase Orders \\Server2.companyABC.local\Files\Finance\Invoices