The Argument for Dynamic Mappings

Oct 13, 2025 by pinkandtrash.com 12 views

Introduction

As developers, we often encounter issues that arise from the intricacies of working with APIs and data storage systems. In this article, we will delve into a specific problem that occurred while working with ElasticSearch, a popular search and analytics engine. We will explore the issue of trailing or starting periods in JSON keys and how it can lead to publisher errors. Additionally, we will discuss potential solutions and best practices for handling such issues.

The Problem

When working with ElasticSearch, it is essential to understand the rules governing JSON keys. In ElasticSearch, JSON keys are not allowed to have periods in the beginning or end of the key. This means that keys like .key or key. are not valid and will result in a publisher error. This issue can arise from various sources, including user input, API requests, or even logging mechanisms.

Inducing the Error

To understand the issue better, let's take a closer look at how it can be induced. In our case, we have a middleware that converts the body and query params into an object on the request map. We also have another middleware that logs all request inputs/outputs for debugging purposes. By curling an endpoint like https://api.example.com?query_param=.query-value, we were able to induce the publisher error.

The Argument for Dynamic Mappings

There is an argument to be made that users should not log arbitrary maps to ES when using dynamic mappings. Dynamic mappings allow for the automatic creation of indices and fields based on the data being indexed. However, this approach can lead to issues like the one we are discussing. On the other hand, logging arbitrary maps can provide valuable insights into the API's input and output data.

Solution and Best Practices

So, how can we avoid this issue? Here are a few solutions and best practices that can help:

1. Validate User Input

One way to avoid this issue is to validate user input before logging it to ElasticSearch. This can be done using a validation library or by implementing custom validation logic. By validating user input, we can ensure that it conforms to the expected format and does not contain any invalid characters.

2. Use a Whitelist Approach

Another approach is to use a whitelist of allowed keys. This means that we only log keys that are explicitly allowed, and ignore any other keys. This approach can help prevent issues like the one we are discussing.

3. Clean the Request Map

We can also clean the request map before logging it to ElasticSearch. This can be done by removing any invalid keys or characters from the map. By cleaning the request map, we can ensure that it conforms to the expected format and does not contain any issues.

4. Use a Custom Logger

Finally, we can use a custom logger that is specifically designed to handle the logging of request inputs and outputs. This logger can be configured to ignore any invalid keys or characters, and can provide valuable insights into the API's input and output data.

Conclusion

In conclusion, the issue of trailing or starting periods in JSON keys can lead to publisher errors in ElasticSearch. By understanding the rules governing JSON keys and implementing solutions like validation, whitelisting, cleaning the request map, and using a custom logger, we can avoid this issue and ensure that our API logs are accurate and reliable. We hope that this article has provided valuable insights into this issue and has helped developers avoid similar problems in the future.

Additional Information

If you find this issue acceptable, we would be happy to contribute it as a PR later this month. We believe that this issue is an important one, and that it can help improve the overall quality and reliability of ElasticSearch logs.

References

ElasticSearch documentation: JSON Keys
ElasticSearch documentation: Dynamic Mappings
ElasticSearch GitHub repository
Clean Keys for Trailing/Starting "." for ElasticSearch Publisher: Q&A ====================================================================

Introduction

In our previous article, we discussed the issue of trailing or starting periods in JSON keys and how it can lead to publisher errors in ElasticSearch. We also explored potential solutions and best practices for handling such issues. In this article, we will provide a Q&A section to address some of the most frequently asked questions related to this topic.

Q: What are the rules governing JSON keys in ElasticSearch?

A: In ElasticSearch, JSON keys are not allowed to have periods in the beginning or end of the key. This means that keys like .key or key. are not valid and will result in a publisher error.

Q: How can I induce the publisher error?

A: You can induce the publisher error by curling an endpoint with a query parameter that contains a period at the beginning or end of the key. For example, https://api.example.com?query_param=.query-value.

Q: What are some potential solutions to this issue?

A: Some potential solutions to this issue include:

Validating user input before logging it to ElasticSearch
Using a whitelist of allowed keys
Cleaning the request map before logging it to ElasticSearch
Using a custom logger that is specifically designed to handle the logging of request inputs and outputs

Q: How can I validate user input in ElasticSearch?

A: You can validate user input in ElasticSearch by using a validation library or by implementing custom validation logic. This can help ensure that user input conforms to the expected format and does not contain any invalid characters.

Q: What is a whitelist approach, and how can I implement it?

A: A whitelist approach involves only logging keys that are explicitly allowed, and ignoring any other keys. You can implement a whitelist approach by creating a list of allowed keys and checking each key against this list before logging it to ElasticSearch.

Q: How can I clean the request map before logging it to ElasticSearch?

A: You can clean the request map by removing any invalid keys or characters from the map. This can be done using a custom function or by using a library that provides a clean function.

Q: What are some best practices for handling this issue?

A: Some best practices for handling this issue include:

Implementing validation and whitelisting mechanisms to prevent invalid keys from being logged
Cleaning the request map before logging it to ElasticSearch
Using a custom logger that is specifically designed to handle the logging of request inputs and outputs
Regularly reviewing and updating the whitelist of allowed keys

Q: Can I contribute this issue as a PR to the ElasticSearch project?

A: Yes, we would be happy to contribute this issue as a PR to the ElasticSearch project. We believe that this issue is an important one, and that it can help improve the overall quality and reliability of ElasticSearch logs.

Conclusion

In conclusion, the issue of trailing or starting periods in JSON keys can lead to publisher errors in ElasticSearch. By understanding the rules governing JSON keys and implementing solutions like validation, whitelisting, cleaning the request map, and using a custom logger, we can avoid this issue and ensure that our API logs are accurate and reliable. We hope that this Q&A article has provided valuable insights into this issue and has helped developers avoid similar problems in the future.

Additional Information

If you have any further questions or concerns, please don't hesitate to reach out. We are always happy to help and provide additional information on this topic.