Skip to main content

Quickstart

Not availableCloud AvailableSelf-Managed Community (OSS)Not availableSelf-Managed Enterprise

Airbyte Open Source is a reliable and extensible open source data pipeline.

If you're getting started with Airbyte Cloud, you can skip ahead to moving data by adding your first source.

This quickstart guides you through creating a locally deployed instance of Airbyte in just minutes using abctl (Airbyte Command Line Tool). You'll be able to move data with minimal setup while you're exploring what Airbyte can do!

If you've already set up an Airbyte instance using Docker Compose and want to move to abctl, see the section on migrating from Docker Compose.

tip

When you're ready to put an Airbyte instance into production, you'll want to review our guides on deployment.

For the best experience, we recommend Deploying Airbyte on Kubernetes via Helm.

On a local deployment, Airbyte's default behavior is to store connector secrets in your configured database. These secrets are stored in plain text and are not encrypted. Refer to the Secret Management documentation to set up an external secrets manager.

If setting up an Airbyte server does not fit your use case needs (i.e. you're using Jupyter Notebooks or iterating on an early prototype for your project) you may find the PyAirbyte documentation useful.

Prerequisites​

  • To use abctl, you'll need to have Docker Desktop installed. See Docker's instructions for installation: Mac, Windows, Linux

1: Install abctl​

The easiest method for installing abctl for Mac and Linux users is to use the following command:

curl -LsfS https://get.airbyte.com | bash -

If you would rather install abctl yourself, follow the instructions for your operating system:

We recommend that Mac users use Brew to install the abctl command.

brew tap airbytehq/tap
brew install abctl

With Brew, you can keep abctl up to date easily, by running:

brew upgrade abctl

2: Run Airbyte​

Ensure that Docker Desktop is up and running. Then, with abctl installed, the following command gets Airbyte running:

tip

By default, abctl only configures an ingress rule for the host localhost. If you plan to access Airbyte outside of localhost, you will need to specify the --host flag to the local install command, providing the FQDN of the host which is hosting Airbyte. For example, abctl local install --host airbyte.company.example.

By specifying the --host flag, Airbyte will be accessible to both localhost and the FDQN passed to the --host flag.

abctl local install

Your browser may open automatically to the Airbyte Application. If not, access it by visiting http://localhost:8000.

You will be asked to enter your email address and an organization name. Your email address will be used to authenticate to your instance of Airbyte. You will also need a password, which is randomly generated as part of the install command. To get your password run:

abctl local credentials

Which should output something similar to:

Credentials:
  Email: user@company.example
  Password: random_password
  Client-Id: 03ef466c-5558-4ca5-856b-4960ba7c161b
  Client-Secret: m2UjnDO4iyBQ3IsRiy5GG3LaZWP6xs9I

Use the value in the password field to authenticate to your new Airbyte instance.

You can set your email and password with the credentials command using abctl. To set your email you can run:

abctl local credentials --email user@company.example

To set your password you can run:

abctl local credentials --password new_password

If you wish to configure authentication when install abctl, follow the documentation on the Authentication Integration page.

As long as your Docker Desktop daemon is running in the background, you can use Airbyte by returning to http://localhost:8000.

If you quit Docker Desktop and want to return to your local Airbyte workspace, just start Docker Desktop again. Once Docker finishes restarting, you'll be able to access Airbyte's local installation as normal.

Suggested Resources​

For the best performance, we suggest you run on a machine with 4 or more CPU's and at least 8 GB of memory. Currently abctl does support running on 2 cpus and 8 gb of ram with the --low-resource-mode flag. You can pass the low resource mode flag when install Airbyte with abctl:

abctl local install --low-resource-mode

Follow this Github discussion to upvote and track progress towards supporting lower resource environments.

3: Move Data​

In the Building Connections section, you'll learn how to start moving data. Generally, there are three steps:

1: Set up a Source

2: Set up a Destination

3: Set up a Connection

Customizing your Installation with a Values file​

Optionally, you can use a values.yaml file to customize your installation of Airbyte. Create the values.yaml on your local storage. Then, apply the values you've defined by running the following command and adjusting the path to the values.yaml file as needed:

abctl local install --values ./values.yaml

Here's a list of common customizations.

Migrating from Docker Compose (Optional)​

note

If you're using an external database or secret manager you don't need to run --migrate flag. You must create the secrets.yaml and values.yaml and then run abctl local install --values ./values.yaml --secret ./secrets.yaml. Please check instructions to setup the external database as example.

If you have data that you would like to migrate from an existing docker compose instance follow the steps below:

  1. Make sure that you have stopped the instance running in docker compose, this may require the following command:
docker compose stop
  1. Make sure that you have the latest version of abctl by running the following command:
curl -LsfS https://get.airbyte.com | bash -
  1. Run abctl with the migrate flag set with the following command:
abctl local install --migrate
note

If you're using a version of Airbyte that you've installed with abctl, you can find instructions on upgrading your Airbyte installation here.

Using an EC2 Instance with abctl​

This guide will assume that you are using the Amazon Linux distribution. However. any distribution that supports a docker engine should work with abctl. The launching and connecting to your EC2 Instance is outside the scope of this guide. You can find more information on how to launch and connect to EC2 Instances in the Get started with Amazon EC2 documentation from Amazon.

  1. Install the docker engine:
sudo yum install -y docker
  1. Add the ec2-user (or whatever your distros default user) to the docker group:
sudo usermod -a -G docker ec2-user
  1. Start and optionally enable (start on boot) the docker engine:
sudo systemctl start docker
sudo systemctl enable docker
  1. Exit the shell and reconnect to the ec2 instance, an example would look like:
exit
ssh -i ec2-user-key.pem ec2-user@1.2.3.4
  1. Download the latest version of abctl and install it in your path:
curl -LsfS https://get.airbyte.com | bash -
  1. Run the abctl command and install Airbyte:
tip

By default, abctl only configures an ingress rule for the host localhost. In order to ensure that Airbyte can be accessed outside of the EC2 instance, you will need to specify the --host flag to the local install command, providing the FQDN of the host which is hosting Airbyte. For example, abctl local install --host airbyte.company.example.

tip

By default, abctl will listen on port 8000. If port 8000 is already in used or you require a different port, you can specify this by passing the --port flag to the local install command. For example, abctl local install --port 6598

Ensure the security group configured for the EC2 Instance allows traffic in on the port (8000 by default, or whatever port was passed to --port) that you deploy Airbyte on. See the Control traffic to your AWS resources using security groups documentation for more information.

abctl local install --host [HOSTNAME]

Running over HTTP​

Airbyte suggest that you secure your instance of Airbyte using TLS. Running over plain HTTP allows attackers to see your password over clear text. If you understand the risk and would still like to run Airbyte over HTTP, you must set Secure Cookies to false. You can do this with abctl by passing the --insecure-cookies flag to abctl:

abctl local install --host [HOSTNAME] --insecure-cookies

Uninstalling​

If you want to remove Airbyte from your system, consider which of the following two options you would like to use.

1: Run the following command to stop all running containers that abctl has created while preserving any data you've created:

abctl local uninstall

2: If you want to clear the persistent data in addition to stopping containers, run:

abctl local uninstall --persisted

As a last step, to clear out any additional information that abctl may have created, you can run:

rm -rf ~/.airbyte/abctl

Troubleshooting​

Using standard tools to interact with an Airbyte instance that was installed with abctl​

abctl install Airbyte into a kind cluster on your local machine. If you'd like to interact directly with any of the underlying infrastructure, you can use standard tooling. You will need to make sure these tools are installed (or install them yourself). Any of these out of the box tools will work with an Airbyte instance installed with abctl.

If you want to interact with the pods or resources inside the cluster you can use kubectl and helm. Just make sure you are pointing at the correct K8s configuration e.g. kubectl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig --namespace airbyte-abctl get pods

kind is a tool for creating a K8s cluster using docker instead of having to install a local K8s cluster. You only need to think about kind if you want to make an adjustment to the cluster itself.

For more advanced interactions (e.g. loading custom docker containers), read more in developing locally.

Unable To Locate User Email​

note

In abctl v0.11.0, support for basic-auth was removed (as basic-auth support was removed from the Airbyte Platform in v0.63.11, and replaced with a more secure randomly generated password. When logging into Airbyte, the email (provided during registration) should be automatically populated. Both the email and the randomly generated password can be fetched by running abctl local credentials.

Airbyte is aware of situations where the email is not be automatically populated and we are working on addressing this within the abctl tool. In the interim, some manually steps are required to retrieve the authentication email address when it is unknown.

If the email address for authenticating is not automatically populated, you can set an email with the following command:

abctl local credentials --email <USER@COMPANY.EXAMPLE>

The password for this user can be retrieved by running abctl local credentials.

Using Custom Connectors​

In order to run a custom connector with an Airbyte instance that is running in kind, you must load the docker image of that connector into the cluster. A connector container can be loaded using the following command:

kind load docker-image <image-name>:<image-tag> -n airbyte-abctl

For more troubleshooting information review the troubleshooting section in Uploading Customer Connectors

Additional Resources​

There are several channels for community support of local setup and deployment.

GitHub Airbyte Forum's Getting Started FAQ:
Search the questions others have asked or ask a new question of your own in the GitHub forum.

Airbyte Knowledge Base:
While support services are limited to Cloud and Enterprise customers, anyone may search the support team's Help Center.

Community Slack:
Helpful channels for troubleshooting include:

  • #ask-community-for-troubleshooting: Where members of the Airbyte community can ask and answer questions.
  • #ask-ai: For quick answers sourced from documentation and open support channels, you can have a chat with our virtual Airbyte assistant.

Introductory Course:
On Udemy, The Complete Hands-on Introduction to Airbyte is a convenient and hands-on introduction to Airbyte that includes setting up example source and destination configurations. You'll also go on to use it in conjunction with Apache Airflow, Snowflake, dbt, and more.

Bug Reports:
If you find an issue with the abctl command, please report it as a github issue here with the type of 🐛 [abctl] Report an issue with the abctl tool.

Releases:
If you'd like to select which release of abctl to run, you can find the list of releases here.

Was this page helpful?