Indexing documents

This guide shows you how to add documents to an index, update existing documents, remove documents, and verify that changes are searchable. For complete API details, see the Documents API.

Prerequisites

Before you index documents, ensure you have:

• A defined index. The index definition must include the fields and primary key used in your documents. If a document includes a field that is not defined in the index, the request is rejected. For more information, see Define an index.
• A running cluster with an available writer node. For more information, see Run more than one node.

Steps

1. Send documents to the index:

   To send a batch of JSON documents, send a POST request to /v1alpha1/indexes/{index_name}/documents:

   POST /v1alpha1/indexes/products/documents
   Content-Type: application/json
   
   {
     "documents": [
       { "id": "1", "name": "Rain jacket", "category": "Outerwear", "price": 129 },
       { "id": "2", "name": "Wool hat", "category": "Hats", "price": 39 }
     ]
   }

   Each document must contain the field defined as the primary key. Sending a document with an existing key replaces the previous document. Indexing reflects desired state, so there is no separate create or update step.

   If you are loading a large dataset, send newline-delimited JSON (application/x-ndjson):

   POST /v1alpha1/indexes/products/documents
   Content-Type: application/x-ndjson
   
   {"id": "1", "name": "Rain jacket", "category": "Outerwear", "price": 129}
   {"id": "2", "name": "Wool hat", "category": "Hats", "price": 39}

   Documents are indexed as the request body is read. Stream the entire file in one request if the request can be reissued, or split it into requests of a few thousand documents if retries should not start from the beginning.

   To send a single document to its own URL, send a PUT request to /v1alpha1/indexes/{index_name}/documents/{key}:

   PUT /v1alpha1/indexes/products/documents/1
   Content-Type: application/json
   
   {
     "name": "Rain jacket",
     "category": "Outerwear",
     "price": 129
   }

   The body can leave the primary key field out. The document is indexed under the key in the URL. The document goes in whole, so a field left out is not kept. To load a dataset, use one of the batch forms above. This form costs one request for each document.

2. Commit changes (optional for bulk loads):

   The index writer commits automatically based on indexed volume or elapsed time (see Committing). If you stream a bulk dataset, send an explicit commit request after all data is loaded:

   POST /v1alpha1/admin/indexes/products/actions/commit

   Send one commit request at the end of the load. Do not commit after every batch, because each commit writes a Lucene commit and pushes it to the remote.

3. Update specific fields (optional):

   To update specific fields in existing documents without sending the entire document, send a POST request to /v1alpha1/indexes/{index_name}/documents/actions/update:

   POST /v1alpha1/indexes/products/documents/actions/update
   
   {
     "documents": [
       { "id": "1", "price": 99, "inStock": true },
       { "id": "2", "discount": null }
     ]
   }

   • A field with a value replaces the current value.
   • A field set to null clears the current value.
   • An omitted field remains unchanged.
   • Locale-specific fields and object fields are replaced whole. To modify part of an object or locale-specific field, send the complete document.
   • If the index definition sets "source": "none", partial updates fail with document:source_not_kept.
   • If a primary key does not exist in the index, the request fails by default. To skip missing keys and receive a list of missing keys in the response, add ?missing=skip to the request URL.

4. Delete documents (optional):

   To delete a single document by its ID, send a DELETE request:

   DELETE /v1alpha1/indexes/products/documents/1

   The server returns 204 whether or not a document exists under that key.

   To delete multiple documents by query, send a POST request:

   POST /v1alpha1/indexes/products/documents/actions/delete
   
   { "query": [ { "field": "category", "match": { "value": "Hats" } } ] }

   Query clauses use the same syntax as the Search API. A query requires at least one clause. To remove all documents and empty the index, send all instead:

   POST /v1alpha1/indexes/products/documents/actions/delete
   
   { "all": true }

Confirming the result

To verify how many documents are searchable in the index, send a search request with limit set to 0:

POST /v1alpha1/indexes/products/search

{ "limit": 0 }

With an empty query, the response returns the total count of searchable documents. Documents indexed since the last commit are not counted until they are committed. If the returned count is lower than expected, commit the index and repeat the search.

The count is what the node answering the search can find. In a deployment with more than one node, a committed change reaches the other nodes on a refresh interval, so a count taken through a load balancer can lag the writer. To count only after the node has the batch, pass the freshness token from the indexing response as freshness.atLeast in the search request. See Make a write visible to search.

Handling errors

If a request fails, use the following guidelines:

• 400 Bad Request: Documents are processed in the order sent. The first refused document fails the request, and the documents sent before it stay in the index. To resolve a failed batch:

  1. Locate the refused document using position in arguments (and line for newline-delimited JSON), and check processed to see how many documents succeeded:

     {
       "code": "validation",
       "errors": [
         {
           "code": "document:field_required",
           "path": "documents[41].name",
           "arguments": { "position": "41", "processed": "41" }
         }
       ]
     }

  2. Fix the refused document and reissue the batch starting from position. Indexing replaces documents by primary key, so resending a document the index already took overwrites it.

  To index valid documents and skip refused entries, add ?onError=skip to the request URL:

  POST /v1alpha1/indexes/products/documents?onError=skip
  Content-Type: application/x-ndjson
  
  {"id": "1", "name": "Blueberry jam"}
  {"id": "2", "nonexistent": "value"}
  {"id": "3", "name": "Rye bread"}

  Inspect the failed array in the response to review skipped entries:

  {
    "indexed": 2,
    "failed": [
      {
        "position": 1,
        "line": 2,
        "errors": [
          {
            "code": "document:field_unknown",
            "message": "Field `nonexistent` does not exist in index",
            "path": "[1].nonexistent"
          }
        ]
      }
    ]
  }

  For the full rules, see Skipping refused entries.

• 409 Conflict: The index has no active writer or is synchronizing. Retry the request.

• 503 Service Unavailable: The index was closed to free disk space. Reissuing the request reopens the index.

Related

• Documents API - Request schemas, response formats, and status codes.
• Updating parts of documents - Change one field, one sub-document, or one locale without resending the rest.
• Defining an index - Define schemas and field validation rules.
• Rolling out a definition change - Reindex documents when an index definition changes.
• Searching an index - Query and retrieve indexed documents.
• Make a write visible to search - The commit and refresh delays between a write and a search that can see it.
• Running multiple nodes - Keeping candidates that can take the writes.
• What a write guarantees - What a 2xx promises, and when a batch is durable.
• Architecture - Why a write reaches the one node that holds the index, and what happens when no node does.