en-US/about_Elastic.Helper.help.txt

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
TOPIC
    about_elastic.helper
 
    SHORT DESCRIPTION
    A PowerShell Helper Module to interface to ElasticSearch
 
    LONG DESCRIPTION
    Interfacing to ElasticSearch is complex. Even with the Elastic.Console
    module. This module solves some problems for me in publishing resources to ElasticSearch.
    This includes:
    * Index Properties
    * Ingest Pipelines
    * Enrichment Policies
    * Bulk Index requests
    This module supports both Authenticated and Unauthenticated connections to
    ElasticSearch. Which mode you use depends on your ElasticSearch implementation.
    This module is not intended to be an exhaustive implementation of
    ElasticSearch functionality. It was built as a way for me to be able to use
    Kibana, etc, to build up a working ElasticSearch configuration to meet a
    specific set of needs, and to be able to reproduce on an additional system.
    The configuration file structure defines high-level resource types, and at
    the resource definition level supports whatever JSON is generated by using Kibana.
    This module leverages the Elastic.Console PowerShell module.
 
    INDEX PROPERTIES
    Index properties can specify which pipeline to use for indexing, as well as
    an ElasticSearch index properties section. This allows you to specify,
    among other things, the number of replicas that should exist for that index,
    as well as any other, configurable index property in ElasticSearch.
 
    INGEST PIPELINES
    An Ingest Pipeline in ElasticSearch defines how the data is to be indexed.
    There are a number of factors that may be used:
    * Field Mappings
    * Date field to use as the primary index date
    * Field to use as Document ID to prevent duplicates
    * Index naming, including date-part
    * Enrichment Policies to apply
    Anything that you can specify in Kibana using the Index Definition can be
    exported as JSON, and saved as the
    definition
    element of the Pipeline in the configuration.
 
    ENRICHMENT POLICIES
    An enrichment policy allows you to add data to an index as it is being
    added. This usually takes the form of a lookup in another index.
    Enrichment policies are, therefore, usually based on an index, and specify
    which fields should be added to the index being eriched. Whenever the base
    index is updated, the Enrichment Policy needs to be refreshed in order for
    the system indices that implement the enrichment lookup to be updated.
    There are some helper functions for this.
    * Rebuild-EsEnrichmentIndices - will rebuild all enrichment indices based on
    the provided configuration
    * Update-EsEnrichmentIndicesFromInxed - will rebuild enrichment indices
    based on a specific index in the provided configuration
 
    BULK INDEX REQUESTS
    Allows for the bulk insert of records to an ElasicSearch index. Currently
    requires the use of Pipelines.
 
    EXAMPLES
    EXAMPLE 1 - DEPLOY CONFIGURATION TO ELASTICSEARCH
    Take the defined configuration: index; pipeline; enrichment; assess any
    dependencies, and deploy the configuration for which dependencies are met to
    ElasticSearch.
 
    Import-Module Elastic.Helper
     
    # Force TLS 1.2
    [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12
     
    $ConfigName = 'elasticproject'
    $CliXmlCreds = '/path/to/clixml.xml'
     
    # Load configuration and credentials
    $EsConfig = Get-EsHelperConfig -ConfigName $ConfigName
    $EsCred = Import-CliXml -Path $CliXmlCreds
     
    Set-ElasticSearchVersion '7.10'
     
    # Deploy non-dependent elements of defined configuration
    Deploy-EsConfig -EsConfig $EsConfig -EsCreds $EsCred
 
    EXAMPLE 2 - BULK INDEX DATA USING SPECIFIED PIPELINE
    Get some SQL data from a database and send it to ElasticSearch via a Bulk
    Index Request.
 
    Import-Module Elastic.Helper
    Import-Module SqlServer
     
    # Force TLS 1.2
    [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12
     
    $ConfigName = 'elasticproject'
    $CliXmlEsCreds = '/path/to/clixmles.xml'
    $CliXmlSqlCreds = '/path/to/clixmlsql.xml'
    $QueryFile = '/path/to/sqlscript.sql'
    $ServerInstance = 'SQLServerName'
     
    # Load configuration and credentials
    $EsConfig = Get-EsHelperConfig -ConfigName $ConfigName
    $EsCred = Import-CliXml -Path $CliXmlEsCreds
     
    Set-ElasticSearchVersion '7.10'
     
    # Get SQL results
    $Data = Invoke-SqlCmd -Credential $CliXmlSqlCreds -ServerInstance $ServerInstance -Database 'Database' -InputFile $QueryFile
     
    # Strip off extraneous elements that cause issues indexing to Elastic
    $DataBare = $Data | Select-Object * -ExcludeProperty ItemArray, Table, RowError, RowState, HasErrors
     
    # Index the data via Bulk Index Request, using the pipeline specified in the $EsConfig definition of the index
    $output = Invoke-EsBulkIndexRequest -EsConfig $EsConfig -IndexName 'myindex' -InputObject $DataBare -EsCreds $LrAieConfig.EsCred -Debug
     
    # Error Handling
    if ($output.errors) {
        $msg = "$(Get-Date) - Warning: Errors encountered.`n {0} Statistics Records synchronised in {1}ms" -f $output.items.Count,$Output.took
        # Extract the items that had errors so you can do something with them
        $output.items.index | Where-Object {$_.status -ge 400}
    } else {
        $msg = "$(Get-Date) - {0} Statistics Records synchronised in {1}ms" -f $output.items.Count,$Output.took
    }
    Write-Output $msg
 
    NOTE
    This module is a work in progress, and is not complete. There are likely to
    be bugs and insufficient error checking/handling.
 
    SEE ALSO
    There will likely be modules releases that leverage this.
    Keep an eye on my GitHub page:
    jberkers42 on GitHub