Importing Data

Note: This guide applies when using ReactiveSearch with Elasticsearch or OpenSearch only.

You can bring your data from various sources into a ReactiveSearch or cluster using one of the following methods:

Dashboard ABC CLI REST API Zapier

This table below can help you choose the right method for your use-case:

Importing Through Dashboard

Dashboard offers a GUI for importing data visually. It supports the following formats:

1. JSON / JSONL - You can bring in any JSON or newline limited JSON files (or hosted URLs) to be imported using the Importer view of the dashboard.
2. CSV - You can also bring in a CSV file (or hosted URL) to be imported.
3. Elasticsearch - You can import an index from an Elasticsearch cluster hosted anywhere into your appbase.io cluster's index.
4. SQL - You can import a SQL database from Postgres, MySQL or Microsoft SQL Server into your appbase.io cluster's index.
5. Shopify - You can import data from a Shopify store into your appbase.io cluster's index. With Shopify, we support options for syncing product, collections, product-collection relationships, meta fields and named tags (a faster alternative to meta fields). Read more about importing from Shopify over here.
   

You can try the Importer UI live by going to Explore Cluster > Develop > Import tab within your dashboard.

Below is a GIF showing the import process for a JSON file.

Importing Through ABC CLI

While the dashboard GUI supports the most common formats, we offer the ABC CLI which allows importing data from an extensive list of database sources using the abc import command.

Key Benefits

• Whether your data resides in Postgres or a JSON file or MongoDB or in all three places, abc can index the data into Elasticsearch. It is the only tool that allows working with all these sources at once or individually: csv, json, postgres, mysql, sqlserver, mongodb, elasticsearch and firestore.
• It can keep the Elasticsearch index synced in realtime with the data source.

  Note: This is currently only supported for MongoDB and Postgres.

• abc import is a single line command that allows doing all of the above. It doesn’t require any external dependencies, code configuration is optional (and helps with transforming data before being indexed), and runs as an isolated process with a minimal resource footprint.

Installation

You can install ABC CLI from the latest Github release for your OS.

You can also install this via docker using:

docker pull appbaseio/abc

It is possible to import data from various database sources. See the snippets below that show the import command for each of the supported database.

Postgres

abc import --src_type=postgres --src_uri=<uri> <elasticsearch_uri>

Read more about it over here.

MongoDB

abc import --src_type=mongodb --src_uri=<uri> <elasticsearch_uri>

Read more about it over here.

MySQL

abc import --src_type=mysql --src_uri=<uri> <elasticsearch_uri>

Read more about it over here.

Microsoft SQL Server

abc import --src_type=mssql --src_uri=<uri> <elasticsearch_uri>

Read more about it over here.

Elasticsearch

abc import --src_type=elasticsearch --src_uri=<elasticsearch_src_uri> <elasticsearch_dest_uri>

Read more about it over here.

JSON

abc import --src_type=json --src_uri=<uri> --typename=<target_type_name> <elasticsearch_uri>

Read more about it over here.

CSV

abc import --src_type=csv --src_uri=<uri> --typename=<target_type_name> <elasticsearch_uri>

Read more about it over here.

Firestore

abc import --src_type=firestore --sac_path=<path_to_service_account_credentials> <elasticsearch_uri>

Read more about it over here.

SQLITE

abc import --src_type=sqlite src_uri=./data.db?_busy_timeout=5000

Read more about it over here.

Kafka

abc import --src_type=kafka src_uri=kafka://user:pass@SERVER:PORT/TOPIC1,TOPIC2

Read more about it over here.

Neo4J

abc import --src_type=neo4j src_uri=bolt://localhost:7687

Read more about it over here.

Redis

abc import --src_type=redis --src_uri="redis://localhost:6379/0" https://USERID:[email protected]/APPNAME

Read more about it over here.

REST API

Appbase.io, by extension of Elasticsearch's REST API offers two ways to index data:

1. Document Index / Document Update endpoint - Read more about this in the doc and see an example code snippet here.
2. Bulk Index endpoint - Read more in the doc and see an example code snippet here.

You can also see the examples below that show an interactive example of using the bulk API for JavaScript, Go, Python and PHP:

Javascript

Go

Python

PHP

Note

It is recommended to index up to 1 MB of data (~500 documents) at a time (so if you have 50,000 documents, you can split them into chunks of 500 documents and index).

Zapier

Appbase.io's Zapier integration allows you to import data from over 1500+ apps such as Mailchimp, Sendgrid, Twilio, Google Sheets, Typeform into your appbase.io app or cluster.

You can start using the integration from here: https://zapier.com/apps/appbaseio/integrations

If you are new to Zapier, it allows connecting apps you use everyday to automate your work and be more productive. You can read more about Zapier over here.

In this section, we will walk through the process of importing data using this Zapier integration. We will use Google Sheets as the input source.

You can also read about importing data from TypeForm into appbase.io using this Zapier integration over here.

Creating A Zap

You can go to the Zapier editor and create a zap. In the below image, we create a zap for Google Sheets <> appbase.io.

Adding Your Data In Google Sheets

Add data in your Google Sheets. You can directly copy the data from here.

Configuring Google Sheets

Login with your Google account and once you configure the Google Sheets integration, you should see something similar:

Configuring appbase.io

Next, select appbase.io from the apps and go to the Create Document action.

After this step, you need to add your API credentials and authenticate. Connect your appbase.io account on clicking Add a New Account under Choose Account section, where you have to enter your appbase.io credentials which you can find here. You should see something similar:

You can perform below operations through Zapier.

Adding New Record

I am going to call my appbase.io app business here. The Zapier integration will create the app if it doesn't exist. The next step is to map the columns from Google Sheets to the field names that will be used to store them in appbase.io. It should look something similar to the following:

After clicking on Continue and after a successful insertion, you will also see an _id being returned back. This is the unique ID for our record (aka document).

Updating An Existing Document

Another helpful feature is the ability to edit existing documents. Whenever a value from an incoming integration is mapped to an _id field, the existing document gets updated instead of a new document getting created.