ElasticSearch is a great tool for full-text search over billions of records. But what if you want to search through files with help of ElastricSearch? How should you extract and index files? After googling for "ElasticSearch searching PDFS", "ElasticSearch index binary files" I didn't find any suitable solution, so I decided to make this post about available options.
During last years we were constantly noticing a bunch of developers and enthusiasts looking for a solution to parse (extract content) and index their documents, such as PDFs. Something they can send or upload their documents into and query them by some phrases from documents' content. Since there was no integrated solution present, they had to improvise. Most of them ended up using Tika or Ingest Attachment plugin along with ElasticSearch, some used Solr or even Lucene with custom parsers. But all these solutions are actually not easy to set up: it usually takes weeks or even months to combine all the modules, fine tune them, fix all bugs and finally get them working together properly (Check out this post: Ingest Attachment Plugin for ElasticSearch: Should You Use It?)
At some point during Ambar development we realized that along with its main goals, it also perfectly solves the issue I described by providing developers with an out-of-the-box solid "parse and search" solution.
Let's discuss a basic scenario of using Ambar for uploading files and search through their contents.
Since the year 2010 we've been creating bespoke security-related software for large enterprises. Almost every enterprise has a security department responsible for background checking of potential customers and partners. And this kind of checking includes regular searching for some info about the subject being checked. The search is performed over hundreds of millions of documents of different kinds and from different sources. Such as reports, articles, financial data, text notes or scanned contracts in pdf, doc, xls, txt or any other formats. Anyway, the issue of fast and efficient searching over huge amount of different docs is quite substantial for any business, especially for large enterprises.
We've gone a long way from using dtSearch to developing our own solution. This post describes our steps and shades some light on why we eventually stopped using dtSearch and created a new solution.
We've been using custom self-created solutions to implement the process of security background checking, but as a full-text search engine we've been using dtSearch for quite a long time (from 2010 to 2016). And that's why:
We are being constantly asked by enthusiasts why can't they simply use Ingest Attachment plugin with ElasticSearch to extract their files content and perform full text search over it. What is so valuable in Ambar that Ingest Attachment + ElasticSearch can't provide you with?
The simplest answer would be:
Just try to process, index and search through a bunch of 500 Mb files with Ingest Attachment + ES and you'll know why :)
But, I'll explain.
Ingest Attachment is a plugin that simply wraps Apache Tika into ES pipeline module that does content and meta extraction out-of-the box. At a first look it might be a silver bullet, the simplest and the most efficient extract-index-search solution. But, it's not as great as it might look like. Let me explain why you should not use it if you're looking for a good quality parsing and going to index and store large amounts of data.
Ok, you signed up for Ambar and uploaded some files, or even connected your Dropbox account. What's next? Ambar is a very powerful searching tool, so let us help you to master it. This post describes the search queries you can use with Ambar.
We're constantly receiving questions: What search engine do you use internally?, How do you collect files?, Why do I need docker to run Ambar?, etc. This post gives you the answers to these questions and clarifies how Ambar works under the hood.
When we were designing Ambar's architecture we were keeping in mind next goals:
You can't reach any of this goals without awesome technologies, below I describe how we achieved each of these goals.
In this post I will tell you my story of digitizing paper work in my home-office. Why do I want to get rid of paper docs? The answer is here.
My way is quite simple and contains of only four steps.
My goal is to create a workflow that will let me efficiently process all my paper documents to eventually end up with a smart digital storage filled up with them. The storage must be secure. Also I want to be able to instantly search for any document that I'm looking for. And, of course, after digitizing, I can not shred the documents and have to store them physically.
What kind of documents I want to digitize? At first, it's my contracts with customers. Yes, it's still done on paper. Second, my personal docs: passport, drivers license, diploma, etc. Third, it's bills, starting from electricity and water, ending with loans.
In this post we will give you 5 bullet proof reasons why you'll benefit from going paperless.
We're continuing our story about creating Ambar, and this is the second paper about ElasticSearch. The first one is Highlighting Large Documents in ElasticSearch. This paper tells the story about making ElasticSearch perform well with documents containing a text field more than 100 Mb in size. The result we achieved is the performance improvement by more than 1100 times compared with the default 'out-of the box' setup.
Ok, let's start!
In December 2016 we started working on Ambar - a documents searching system. Ambar uses ElasticSearch as a core search engine.
During the process of Ambar development we dealt with many issues related to ES, and we'd like to share the precious experience we gained. Let's start with a substantial function of every search system - search results highlighting.
Proper results highlighting is the most valuable part of any search system's usability, at first, it provides a user with the necessary info about internal search logic and why the result was retrieved. Also it makes it possible to estimate results only by a quick glance at highlights instead of downloading and exploring the whole document.
