Scott's Recipes Logo

Building Database Level Multitenancy into Rails 8 From Scratch


Please note that all opinions are that of the author.


Last Updated On: 2025-11-26 06:51:47 -0500

Disclaimer: This is, as of 11/16/25, still a work in progress. Most of this works but it has failed at the deploy stage. It is still being published because I need it in a browser so I can switch between write, working code in an editor and an attempt to make it into a template for testing in a second editor (Brave / TextMate 2 / Sublime Text 2). Use with care.

Multitenancy, the storage of different customers data isolated from each other, is not a Rails specialty. Rails is very much oriented towards “the majestic monolith” aka Basecamp where there is a single database and a single shared connection (or pool of connections) to that database. However, for lots of applications, multitenancy is desirable. There are two common approaches to multitenancy:

In this blog post, I am talking about the second case – Database Level Multitenancy. I just completed a long, multi day pass at converting our application to this style of tenancy and I thought it useful to write all of this, step by step.

Note: This is written from work in progress code and there may well be some cruft below. The intent in writing this before it was fully completed was to force a design review while all details were still fresh in my mind.

Step 0: Work in a Branch

This process, honestly, is going to suck. So make a branch and work exclusively in it. Yes you should always do that but lots of times you don’t scope things correctly and work in main. Don’t be that developer. This is bigger than a bread basket so work in a branch.

Also please know that Rails 8 changed a lot of the underlying internal ActiveRecord connection details so documentation and gems prior to Rails 8 will be wrong or not work.

Step 1: Understand the Scope of What You Are Doing

At its very nature, Rails DOES NOT WANT YOU TO DO THIS and that means you have to mess with a lot of low level aspects of Rails itself. This will have ongoing maintenance costs so keep this in mind.

Here are the technical hurdles you have to get past:

  1. A separate registry that makes incoming domains making requests to database names.
  2. Your own migration facility that runs migrations against each tenants.
  3. A set of rake tasks which handle tenant creation, migration running, tenant dropping, debugging, etc.
  4. An around filter in ApplicationController to connect to the right database.
  5. Turning off the default “you can’t do anything because migrations exist” warning.
  6. An abstract class to handle connecting to the database.

Making all these changes involved fewer files than you would expect. Here is my current git status from working on this for about 4 to 5 days. Please note that there are some scratch files that you likely won’t have.

❯ git status
On branch hippie
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   app/controllers/application_controller.rb
	modified:   app/models/tenant.rb
	modified:   config/application.rb
	modified:   config/database.yml
	modified:   config/environments/development.rb
	modified:   config/environments/production.rb
	modified:   lib/tasks/tenants.rake
	modified:   test/models/tenant_test.rb
	modified:   test/test_helper.rb

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	app/models/temp_connection.rb
	app/models/tenant_base.rb
	app/models/tenant_connection.rb
	app/models/tenant_registry.rb
	bin/move_migrations
	bin/move_migrations_back
	lib/tasks/diagnose_tenants.rake
	test/models/tenant_connection_test.rb

It is a truism of all development technologies that there is no such thing as a free lunch (the TANSTAAFL principle). As an example, back in the 90s we had easy to use integrated debugging in environments like Visual Basic that are STILL superior to debugging on the web. We gave up easy debugging in exchange for world wide web style ubiquity. With this approach to multi tenancy, we give up some of the conveniences of Rails like the automatic prompting to run database migrations in exchange for database scalability. And, yes, this will come back to bite us at some point but it is a conscious choice on my part. And, perhaps, we can figure out a way to address it in the future.

Step 2: Configuration Changes

You need to make configuration changes to these files:

Local Development with Host Names

My application is called polly and it runs (sometimes) on top of the pollitify.com domain name. Given that pollitify itself is a separate Rails app, I run two apps, one on port 3000 and one on port 3500. The url structure I needed to emulate was:

The way to get this url structure in Rails is to modify application.rb with a block of code like this:

         if Rails.env.development?
           config.hosts = []
           config.hosts << "polly.pollitify:3000"
         end

This goes inside this structure:

    module Polly
      class Application < Rails::Application

It is important to note that there is a second step to all this: YOU HAVE TO ADD THE NAMES TO /etc/hosts. You want to edit:

sudo nano /etc/hosts

And your contents might look sometime like this:

# allows me to goto http://pollitify:3500 for my other Rails app
127.0.0.1       pollitify
# allows me to goto http://polly.pollitify:3000 for this rails app
# will run on https://polly.pollitify.com/ in production
127.0.0.1       polly.pollitify
# allows me to goto http://soupforourfamilies:3000 for this rails app
# will run on https://soupforourfamilies.org/ in production
127.0.0.1       soupforourfamilies
# allows me to goto http://helpfulhippie:3000 for this rails app
# will run on https://www.helpfulhippie.com/ in production
127.0.0.1       helpfulhippie

NOTE: If you are doing Rails development under windows without the WSL then I have no clue how to do this but go wild; you’ll figure it out.

As with all changes to config/application.rb, the whole application needs to be restarted for the changes to take effect. So go to the terminal window where you run bin/dev and ctrl+c it and then restart it.

It Works!!!

After I wrote this section, I went in my browser to:

http://helpfulhippie:3000

And I got this error:

Tenant not found
Extracted source (around line #48):
          
  def switch_tenant
    tenant = Tenant.find_by(domain: Tenant.strip_www(request.host))
    raise ActiveRecord::RecordNotFound, "Tenant not found" unless tenant
    # Dynamically establish the connection
    tenant_config = ActiveRecord::Base.configurations.configs_for(env_name: Rails.env, name: "primary")

My next step was to do this:

bundle exec rake tenants:list –trace

** Invoke tenants:list (first_time) ** Invoke environment (first_time) ** Execute environment ** Execute tenants:list

=== Tenants === ✓ helpfulhippie.com → polly_helpfulhippie_com_development ✓ polly.pollitify → polly_polly_pollitify_development ✓ soupforourfamilies → polly_soupforourfamilies_development ✓ soupforourfamilies.pollitify → polly_soupforourfamilies_pollitify_development

So a local development tenant was, in fact, missing. This led me to one of the rake tasks:

bundle exec rake tenants:create\[helpfulhippie,HelpfulHippie\]

The first parameter is the host name and since this is local, it will just be “helpfulhippie” and the second parameter is the “customer” name i.e. just something that your billing engine needs (sidebar: I may ultimately eliminate that and just use the domain name).

The result of this is:

bundle exec rake tenants:create\[helpfulhippie,HelpfulHippie\]

=== Creating New Tenant ===
Domain: helpfulhippie
Company: HelpfulHippie
✓ Tenant record created
  ID: 19
  Database: polly_helpfulhippie_development

→ Creating database: polly_helpfulhippie_development...
Created database 'polly_helpfulhippie_development'
✓ Database created

→ Running migrations...
✓ Connected to: polly_helpfulhippie_development
→ Running 20250222140200_devise_create_users.rb...
==  DeviseCreateUsers: migrating ==============================================
-- create_table(:users)
   -> 0.0047s
-- add_index(:users, :email, {unique: true})
   -> 0.0012s
-- add_index(:users, :username, {unique: true})
   -> 0.0013s
-- add_index(:users, :reset_password_token, {unique: true})
   -> 0.0012s
-- add_index(:users, :confirmation_token, {unique: true})
   -> 0.0011s
-- add_index(:users, :unlock_token, {unique: true})
   -> 0.0013s
==  DeviseCreateUsers: migrated (0.0110s) =====================================

... (omitted for sake of brevity)

→ Running 20251102102133_add_published_at_to_pages.rb...
==  AddPublishedAtToPages: migrating ==========================================
-- add_column(:pages, :published_at, :datetime)
   -> 0.0014s
==  AddPublishedAtToPages: migrated (0.0014s) =================================

→ Running 20251102122408_add_category_index_to_pages.rb...
==  AddCategoryIndexToPages: migrating ========================================
-- add_index(:pages, [:category, :published_at], {name: "category_published_at"})
   -> 0.0021s
==  AddCategoryIndexToPages: migrated (0.0021s) ===============================


✓ Tenant created successfully
  Domain: helpfulhippie
  Database: polly_helpfulhippie_development
  Company: HelpfulHippie

What you can see from the above is the following:

  1. When you first when to http://helpfulhippie:3000, no tenant was found so an error was raise.
  2. When you create the tenant, the migrations kick off automatically and migrate the entire schema

So the operational implication of this from the low level aspects of running a SAAS business is:

  1. Sign up a new customer
  2. Log into the production environment
  3. Run the create tenant routine
  4. A new instance of the Polly software is available to the customer

Sidebar 1: Is This Scalable Operationally?

There is clearly an implication here that this isn’t a scalable approach however, there is absolutely NO REASON that you couldn’t write an admin tool in front of this and then run these migrations via a Sidekiq style deferred job. I simply haven’t done this (yet). We are at the early stages of launching this business and we do not need to scale it yet.

Sidebar 2: Security Considerations

Given that the registry database is a component of the overall multitenancy, access to changing that does need to be absolutely locked down. There is an argument to NOT having a web ui because ssh access to the production hosting environment is a way of guaranteeing that lock down. And while that isn’t a long term strategy, it will, in fact, work for the short term.

Sidebar 3: Persistent File Storage

For a number of operational reasons, we have opted to provide persistent file storage NOT through cloud providers but through persistently attached volumes to our physical servers. There is a need to create a master directory below the mount point where each tenant’s files can live. Because this happens OUTSIDE of the Docker file system, there isn’t an easy way to automate that initially (ultimately there will be a small http style api outside of the Docker world to create that) but, initially, it will be a manual action.

Here are examples of the mkdir commands that were ran to support these initial tenants:

root@pollitify01:~# mkdir /mnt/uploads/polly/polly.pollitify.com
root@pollitify01:~# mkdir /mnt/uploads/polly/helpfulhippie.com
root@pollitify01:~# mkdir /mnt/uploads/polly/soupforourfamilies.org

Sidebar 4: Scalability

One of the ways that you provide multitenant scalability with this style architecture is by moving databases from a database server which is at capacity to a database server which has capacity. While this isn’t supported today, it could easily be handled by adding a database_server field which connected to a different postgres server.

config/database.yml

Although you are making Rails multitenant, you will still rely on the good, old database.yml for one thing: your connection to the registry database of tenants. Here’s how to do that:

development:
  primary:
    <<: *default
    database: polly_polly_pollitify_development
    host: localhost
  registry:
    <<: *default
    database: polly_registry_development
    host: localhost

test:
  primary:
    <<: *default
    database: polly_polly_pollitify_test
    host: localhost
  registry:
    <<: *default
    database: polly_registry_test
    host: localhost
  tenant_test_db:
    <<: *default
    database: tenant_test_db

config/environments/development.rb and config/environments/production.rb

In both these file syou need to add this line:

    # comment this next line out if it exists for you
    # config.active_record.migration_error = :page_load
    # Disable automatic pending migration check in development
    config.active_record.migration_error = false

You need to do this for both development and production.

Step 3: The Registry Database

Given that you have to choose the right database to connect to from an incoming hostname, you need a separate database to store this information.

Create the directory for migrations

Start by creating the directory to store the migrations:

mkdir db/registry_migrate

Here is my migration for this:

cat db/registry_migrate/20250107000001_create_tenants.rb
    class CreateTenants < ActiveRecord::Migration[8.0]
      def change
        create_table :tenants do |t|
          t.string :domain, null: false
          t.string :database_name, null: false
          t.string :company_name
          t.boolean :active, default: true
          t.json :settings
          t.timestamps
        end

        add_index :tenants, :domain, unique: true
        add_index :tenants, :database_name, unique: true
      end
    end

Even though the registry database migration is directly and specifically tied to the tenant model, I am covering that in the next section due to its complexity.

Step 4: The Tenant Model

The Tenant class creates the underlying registry which maps host names to database names:

class Tenant < TenantConnection #TempConnection #< ApplicationRecord
  
  validates :domain, presence: true, uniqueness: true
  validates :database_name, presence: true, uniqueness: true
  
  before_validation :set_database_name, on: :create
  
  # Example method that connects to the tenant’s DB for a given block
  def with_connection
    config = ActiveRecord::Base.configurations.configs_for(env_name: Rails.env, name: database_name)
    raise "No config found for #{database_name}" unless config

    ActiveRecord::Base.connected_to(database: { writing: database_name }) do
      yield
    end
  end
  
  # Tenant.get_database_name_from_host("polly.pollitify")
  def self.get_database_name_from_host(domain)
    "polly_#{Tenant.strip_www(domain).gsub(/[^a-z0-9]/i, '_').downcase}_#{Rails.env}"
  end
  
  def self.strip_www(domain)
    domain.sub(/^www./,'')
  end
  
  def switch!
    TenantConnection.switch!(database_name)
  end

  def connection_config
    if Rails.env.production?
      # In production, build config from DATABASE_URL pattern
      # but override the database name
      base_config = ActiveRecord::Base.connection_db_config.configuration_hash
      base_config.merge(database: database_name)
    else
      # In development/test, use explicit config
      Rails.configuration.database_configuration[Rails.env]['primary']
        .merge('database' => database_name)
    end
  end

  private

  def set_database_name
    # Convert domain to safe database name
    # e.g., "client.example.com" -> "polly_client_example_com_development"
    self.database_name ||= Tenant.get_database_name_from_host(domain)
  end
end

The Tenant class inherits from the abstract class TenantConnection. This is a new Rails 8 thing where database connection stuff has to live in an abstract class.

class TenantConnection < ActiveRecord::Base
  self.abstract_class = true

  # All models inheriting from this connect to the registry by default
  connects_to database: { writing: :registry, reading: :registry }
  
  # Switch to a tenant or named database dynamically
  #
  # Works for:
  #   TenantConnection.switch!("registry")     # from database.yml
  #   TenantConnection.switch!("tenant_123")   # dynamic tenant
  #   TenantConnection.switch!(hash_config)    # full hash
  #
  def self.switch!(config_or_name)
    config =
      case config_or_name
      when String, Symbol
        lookup_config(config_or_name.to_s) || build_dynamic_config(config_or_name.to_s)
      when Hash
        config_or_name
      else
        raise ArgumentError, "Invalid config_or_name: #{config_or_name.inspect}"
      end

    raise ArgumentError, "No configuration found for #{config_or_name}" unless config

    establish_connection(config)
    connection  # <-- return actual connection
  end

  # Look up a named connection from database.yml 
  def self.lookup_config(name)
    configurations = ActiveRecord::Base.configurations
    return nil unless configurations.respond_to?(:configs_for)

    configs = configurations.configs_for(env_name: Rails.env, name: name)
    db_config =
      case configs
      when ActiveRecord::DatabaseConfigurations::HashConfig
        configs
      when Array
        configs.first
      end

    db_config&.configuration_hash
  rescue => e
    warn "lookup_config failed for #{name}: #{e.message}"
    nil
  end

  # Build a runtime tenant configuration when it's not in database.yml
  def self.build_dynamic_config(db_name)
    configurations = ActiveRecord::Base.configurations
    primary = configurations.configs_for(env_name: Rails.env, name: "primary")

    # Handle both Array and single HashConfig return types
    primary_config =
      case primary
      when ActiveRecord::DatabaseConfigurations::HashConfig
        primary
      when Array
        primary.first
      else
        nil
      end

    # Fallback to the environment’s top-level config if no primary exists
    unless primary_config
      env_configs = configurations.configs_for(env_name: Rails.env)
      primary_config =
        case env_configs
        when ActiveRecord::DatabaseConfigurations::HashConfig
          env_configs
        when Array
          env_configs.first
        else
          nil
        end
    end

    raise "Primary database configuration not found" unless primary_config

    config = primary_config.configuration_hash.deep_dup
    config[:database] = db_name
    config
  end

  # Optionally helpful for debugging
  def self.current_database
    connection_db_config.database
  end

  def self.clear!
    connection_pool.disconnect!
    establish_connection(:primary)
    true
  end

end


Of course there should be at least some test coverage:

# test/models/tenant_test.rb
require "test_helper"

class TenantTest < ActiveSupport::TestCase
  use_registry
  
  test "creates tenant with valid domain" do
    tenant = Tenant.create!(domain: "test.example.com")
    assert_equal "polly_test_example_com_test", tenant.database_name
  end
  
  test "validates uniqueness of domain" do
    Tenant.create!(domain: "test.example.com")
    duplicate = Tenant.new(domain: "test.example.com")
    assert_not duplicate.valid?
  end
  
  test "gets the domain name from the host name for fully qualified domain name" do 
    db_name = Tenant.get_database_name_from_host("helpfulhippie.com")
    assert_equal "polly_helpfulhippie_com_test", db_name
  end
  
  test "gets the domain name from the host name for fully qualified domain name and deals with www correctly" do 
    db_name = Tenant.get_database_name_from_host("www.helpfulhippie.com")
    assert_equal "polly_helpfulhippie_com_test", db_name
  end
  
  test "gets the domain name from the host name for a subdomain" do 
    db_name = Tenant.get_database_name_from_host("cassjackson.47muppet.com")
    assert_equal "polly_cassjackson_47muppet_com_test", db_name
  end
end

And we need some coverage on TenantConnection so:

# test/models/tenant_connection_test.rb
require "test_helper"

# test/models/tenant_connection_test.rb
require "test_helper"

class TenantConnectionTest < ActiveSupport::TestCase
  self.use_transactional_tests = false
  
  setup do
    @tenant_name = "tenant_test_db_for_test"

    # Ensure we start with a clean database
    drop_tenant_db(@tenant_name) rescue nil
    create_tenant_db(@tenant_name)

    # Minimal tenant config pointing to the real Postgres DB
    @tenant_config = {
      "adapter"  => "postgresql",
      "database" => @tenant_name,
      "username" => ENV['PGUSER'] || "postgres",
      "password" => ENV['PGPASSWORD'] || nil,
      "host"     => ENV['PGHOST'] || "localhost",
      "port"     => ENV['PGPORT'] || 5432
    }

    @hash_config = ActiveRecord::DatabaseConfigurations::HashConfig.new(
      Rails.env, @tenant_name, @tenant_config
    )
    
    @registry_config = ActiveRecord::DatabaseConfigurations::HashConfig.new(
      Rails.env,
      "registry",
      {
        "adapter"  => "postgresql",
        "database" => "registry_test_db",
        "username" => ENV['PGUSER'] || "postgres",
        "password" => ENV['PGPASSWORD'] || nil,
        "host"     => ENV['PGHOST'] || "localhost",
        "port"     => ENV['PGPORT'] || 5432
      }
    )
  end

  teardown do
    drop_tenant_db(@tenant_name) rescue nil
  end

  test "switch! allows inserts and queries on the tenant database" do
    with_real_db_configuration([@registry_config, @hash_config]) do
      # Switch to the tenant DB
      TenantConnection.switch!(@hash_config.configuration_hash)
      
      # Use a temporary model to test inserts
      klass = Class.new(ActiveRecord::Base) do
        self.table_name = "temp_test_table"
      end

      # Create a table in the tenant DB
      klass.connection.create_table(:temp_test_table) do |t|
        t.string :name
      end

      # Insert a record
      klass.create!(name: "Test Record")

      # Query it back
      record = klass.first
      assert_equal "Test Record", record.name

      # Drop the table to clean up
      klass.connection.drop_table(:temp_test_table)
    end
  end

  private

  # Temporarily overrides configurations to return a real DB config
  def with_real_db_configuration(configs)
    original = ActiveRecord::Base.configurations
    fake = StubConfigs.new(configs)
    ActiveRecord::Base.singleton_class.prepend(Module.new {
      define_method(:configurations) { fake }
    })
    yield
  ensure
    ActiveRecord::Base.singleton_class.prepend(Module.new {
      define_method(:configurations) { original }
    })
  end

  # Helpers to create/drop tenant databases
  def create_tenant_db(name)
    ActiveRecord::Base.establish_connection(Rails.env.to_sym)
    ActiveRecord::Base.connection.create_database(name)
  end

  def drop_tenant_db(name)
    ActiveRecord::Base.establish_connection(Rails.env.to_sym)
    ActiveRecord::Base.connection.drop_database(name)
  end
end

# Minimal stub class to mimic Rails configurations
class StubConfigs
  def initialize(configs)
    @configs = configs
  end

  # Rails calls this to get a list of matching configs
  def configs_for(env_name:, name:)
    @configs
  end
  
  # Rails 8 sometimes calls this to check if a config set represents the primary database
  def primary?(*)
    false
  end

  # Rails 8 calls this with either:
  # - a HashConfig (already resolved)
  # - a Hash (legacy style)
  # - a String or Symbol (new behavior)
  def resolve(config)
    case config
    when ActiveRecord::DatabaseConfigurations::HashConfig
      config
    when Hash
      ActiveRecord::DatabaseConfigurations::HashConfig.new(
        Rails.env,
        config["database"] || "stub_db",
        config
      )
    when String, Symbol
      found = @configs.find { |c| c.name.to_s == config.to_s }
      if found
        found
      else
        raise "No configuration found for #{config.inspect}"
      end
    else
      raise "Unexpected config type: #{config.class.name}"
    end
  end
end

These lines need to go into test_helper.rb:

class ActiveSupport::TestCase
  include Devise::Test::IntegrationHelpers
  
  def setup
    setup_registry if self.class.use_registry?
  end
  
  def self.use_registry
    @use_registry = true
  end
  
  def self.use_registry?
    @use_registry
  end
  
  private
  
  # Setup registry database before tests
  def setup_registry
    TenantConnection.connection    
  end  
end

The test_helper.rb file originally had registry creation inside it but making that work consistently and reliably was awful. Now I have extracted into a bash script that calls the entire test preparation process and then runs a rake task that builds the registry.

bin/test_setup

#!/bin/bash
ECHO "About to setup for test environment"
RAILS_ENV=test bin/rails db:create
RAILS_ENV=test bin/rails db:test:prepare
RAILS_ENV=test bin/rails db:create:registry
RAILS_ENV=test bin/rails test:prepare_registry

test.rake


namespace :test do
  desc "Prepare registry database for testing"
  task prepare_registry: :environment do
    Rails.env = "test"
    
    # Get the registry database configuration
    db_config = ActiveRecord::Base.configurations.configs_for(env_name: "test", name: "registry")
    
    # Create the database if it doesn't exist
    ActiveRecord::Tasks::DatabaseTasks.create(db_config)
    
    # Establish connection to registry
    ActiveRecord::Base.establish_connection(db_config)
    
    # Ensure schema_migrations table exists
    unless ActiveRecord::Base.connection.table_exists?(:schema_migrations)
      ActiveRecord::Base.connection.create_table :schema_migrations, id: false do |t|
        t.string :version, null: false, primary_key: true
      end
    end
    
    # Run migrations from the registry migrations folder
    migrations_path = Rails.root.join("db/registry_migrate")
    
    if migrations_path.exist?
      puts "Running migrations from #{migrations_path}"
      ActiveRecord::Migration.verbose = true
      
      # Get all migration files
      migration_files = Dir[migrations_path.join("*.rb")].sort
      
      migration_files.each do |file|
        version = File.basename(file).split('_').first
        
        # Check if already migrated
        unless ActiveRecord::Base.connection.select_values("SELECT version FROM schema_migrations").include?(version)
          puts "Running migration #{File.basename(file)}"
          load file
          
          # Get the migration class name from the file
          migration_name = File.basename(file, '.rb').split('_')[1..-1].join('_').camelize
          migration_class = migration_name.constantize
          
          migration_class.new.migrate(:up)
          
          # Record migration
          ActiveRecord::Base.connection.execute("INSERT INTO schema_migrations (version) VALUES ('#{version}')")
        end
      end
    end
    
    # Disconnect
    ActiveRecord::Base.connection_pool.disconnect!
  end
end

Step 5: ApplicationController

All of the above code means absolutely nothing without the ability to, at runtime, select the right database and connect to it. This is done using ApplicationController and an around filter:

class ApplicationController < ActionController::Base
  
  #around_action :switch_tenant_database
  around_action :switch_tenant, unless: -> { Rails.env.test? }
  
  private
  
  def switch_tenant
    tenant = Tenant.find_by(domain: Tenant.strip_www(request.host))
    raise ActiveRecord::RecordNotFound, "Tenant not found" unless tenant

    # Dynamically establish the connection
    tenant_config = ActiveRecord::Base.configurations.configs_for(env_name: Rails.env, name: "primary")

    db_config = tenant_config.configuration_hash.merge(
      "database" => tenant.database_name
    )

    ActiveRecord::Base.establish_connection(db_config)

    yield
  ensure
    # Reset back to registry after request
    ActiveRecord::Base.establish_connection(:primary)
  end
  
end

Please note that the around_action :switch_tenant, unless: -> { Rails.env.test? } means that you technically are NOT testing in a tenant context. Fixing this isn’t impossible but it feels harder right now than I have the energy for. Right now I need to get back to feature development so it is being left as an outstanding issue.

Step 6: Doing this in Console or Rake Tasks

Any experienced Rails developer, whether they want to admit it or not, knows that sometimes you have you log into the database and do something manually. Here’s how to do that:

Please note that this is TBD at present. It needs to be done but I honestly don’t have the mental energy to touch it at present.

Step 7: The tenants.rake task

This following code is probably the worst bit of all of this. A lot of this should be refactored into testable methods but, honestly, I just want to take a nap at this point. A refactoring will happen not yet.

# lib/tasks/tenants.rake

=begin
rake tenants:create[domain,company_name]                 # Create a new tenant database
rake tenants:create_from_legacy[domain]                  # Create initial tenant from legacy database
rake tenants:drop[domain]                                # Drop a tenant database (DANGEROUS)
rake tenants:list                                        # List all tenants
rake tenants:migrate                                     # Run migrations for all tenant databases
rake tenants:migrate_registry                            # Migrate registry database
rake tenants:seed                                        # Run custom seed for all tenants
rake tenants:seed_one[domain]                            # Run custom seed for specific tenant
rake tenants:setup_registry                              # Setup registry database

=end

namespace :tenants do
  # DONE AND TESTED
  # bundle exec rake tenants:setup_registry --trace
  desc "Setup registry database"
  task setup_registry: :environment do
    registry_config = get_registry_config
    
    puts "\n=== Registry Database Setup ==="
    puts "Database: #{registry_config['database'] || registry_config['url']}"
    
    puts "\nCreating registry database..."
    begin
      ActiveRecord::Tasks::DatabaseTasks.create(registry_config)
      puts "✓ Registry database created"
    rescue ActiveRecord::DatabaseAlreadyExists
      puts "✓ Registry database already exists"
    end
    
    puts "\nRunning registry migrations..."
    
    original_config = ActiveRecord::Base.connection_db_config
    
    begin
      puts "→ Connecting to: #{registry_config['database']}"
      ActiveRecord::Base.establish_connection(registry_config)
      
      # Verify connection
      connection = ActiveRecord::Base.connection
      puts "✓ Connected to database: #{connection.current_database}"
      
      # Ensure migrations directory exists
      migrations_path = Rails.root.join("db/registry_migrate")
      puts "→ Registry migrations path: #{migrations_path}"
      FileUtils.mkdir_p(migrations_path) unless Dir.exist?(migrations_path)
      
      # Check for migration files
      migration_files = Dir[migrations_path.join("*.rb")].sort
      puts "→ Found #{migration_files.count} migration file(s)"
      migration_files.each { |f| puts "  - #{File.basename(f)}" }
      
      # Create schema_migrations table if it doesn't exist
      unless connection.table_exists?(:schema_migrations)
        puts "→ Creating schema_migrations table..."
        connection.execute(<<-SQL)
          CREATE TABLE IF NOT EXISTS schema_migrations (
            version varchar NOT NULL PRIMARY KEY
          );
        SQL
        puts "✓ Schema migrations table created"
      end
      
      # Create ar_internal_metadata table if it doesn't exist
      unless connection.table_exists?(:ar_internal_metadata)
        puts "→ Creating ar_internal_metadata table..."
        connection.execute(<<-SQL)
          CREATE TABLE IF NOT EXISTS ar_internal_metadata (
            key varchar NOT NULL PRIMARY KEY,
            value varchar,
            created_at timestamp(6) NOT NULL,
            updated_at timestamp(6) NOT NULL
          );
        SQL
        
        connection.execute(<<-SQL)
          INSERT INTO ar_internal_metadata (key, value, created_at, updated_at)
          VALUES ('environment', '#{Rails.env}', NOW(), NOW())
          ON CONFLICT (key) DO NOTHING;
        SQL
        puts "✓ Internal metadata table created"
      end
      
      # Get already-run migrations
      migrated_versions = connection.select_values("SELECT version FROM schema_migrations")
      puts "→ Already migrated: #{migrated_versions.count} version(s)"
      
      # Run pending migrations manually
      migration_files.each do |migration_file|
        # Extract version from filename (e.g., "20250107000001_create_tenants.rb" -> "20250107000001")
        version = File.basename(migration_file).split('_').first
        
        if migrated_versions.include?(version)
          puts "  ⊘ Skipping #{File.basename(migration_file)} (already migrated)"
          next
        end
        
        puts "  → Running #{File.basename(migration_file)}..."
        
        begin
          # Load the migration file
          load migration_file
          
          # Extract the class name from the filename
          # e.g., "20250107000001_create_tenants.rb" -> "CreateTenants"
          class_name = File.basename(migration_file, '.rb').split('_')[1..-1].map(&:capitalize).join
          
          # Get the migration class
          migration_class = class_name.constantize
          
          # Run the migration
          migration_instance = migration_class.new
          migration_instance.migrate(:up)
          
          # Record the migration
          connection.execute("INSERT INTO schema_migrations (version) VALUES ('#{version}')")
          
          puts "  ✓ Migrated #{File.basename(migration_file)}"
        rescue => e
          puts "  ✗ Failed to run #{File.basename(migration_file)}"
          puts "  Error: #{e.class} - #{e.message}"
          raise e
        end
      end
      
      puts "✓ Registry setup complete"
    rescue => e
      puts "\n✗ Error during registry migration"
      puts "Database: #{registry_config['database']}"
      puts "Error: #{e.class} - #{e.message}"
      puts "\nBacktrace:"
      puts e.backtrace.first(15).map { |line| "  #{line}" }.join("\n")
      exit 1
    ensure
      ActiveRecord::Base.establish_connection(original_config.configuration_hash)
    end
  end
















  # DONE AND TESTED
  # bundle exec rake tenants:migrate_registry --trace
  desc "Migrate registry database"
  task migrate_registry: :environment do
    registry_config = get_registry_config
    
    puts "\n=== Migrating Registry Database ==="
    puts "Database: #{registry_config['database'] || registry_config['url']}"
    
    original_config = ActiveRecord::Base.connection_db_config
    
    begin
      ActiveRecord::Base.establish_connection(registry_config)
      connection = ActiveRecord::Base.connection
      puts "✓ Connected to: #{connection.current_database}"
      
      migrations_path = Rails.root.join("db/registry_migrate")
      migration_files = Dir[migrations_path.join("*.rb")].sort
      
      # Get already-run migrations
      migrated_versions = connection.select_values("SELECT version FROM schema_migrations")
      
      # Run pending migrations
      migration_files.each do |migration_file|
        version = File.basename(migration_file).split('_').first
        
        if migrated_versions.include?(version)
          puts "⊘ Skipping #{File.basename(migration_file)}"
          next
        end
        
        puts "→ Running #{File.basename(migration_file)}..."
        
        load migration_file
        class_name = File.basename(migration_file, '.rb').split('_')[1..-1].map(&:capitalize).join
        migration_class = class_name.constantize
        migration_instance = migration_class.new
        migration_instance.migrate(:up)
        
        connection.execute("INSERT INTO schema_migrations (version) VALUES ('#{version}')")
        puts "✓ Migrated #{File.basename(migration_file)}"
      end
      
      puts "✓ Registry migrated"
    rescue => e
      puts "\n✗ Error migrating registry"
      puts "Database: #{registry_config['database']}"
      puts "Error: #{e.class} - #{e.message}"
      puts "\nBacktrace:"
      puts e.backtrace.first(15).map { |line| "  #{line}" }.join("\n")
      exit 1
    ensure
      ActiveRecord::Base.establish_connection(original_config.configuration_hash)
    end
  end











  # bundle exec rake tenants:migrate--trace
  desc "Run migrations for all tenant databases"
  task migrate: :environment do
    puts "\n=== Migrating All Tenant Databases ==="
    
    tenants = Tenant.where(active: true).to_a
    puts "Found #{tenants.count} active tenant(s)\n"
    
    migrated = 0
    failed = 0
    failed_tenants = []
    
    migrations_path = Rails.root.join("db/migrate")
    migration_files = Dir[migrations_path.join("*.rb")].sort
    
    tenants.each do |tenant|
      puts "\n" + "="*60
      puts "→ Tenant: #{tenant.domain}"
      puts "  Database: #{tenant.database_name}"
      puts "="*60
      
      begin
        connection = TenantConnection.switch!(tenant.database_name)
        #connection = ActiveRecord::Base.connection
        current_db = connection.current_database
        puts "  ✓ Connected to: #{current_db}"
        
        unless current_db == tenant.database_name
          raise "Database mismatch! Expected #{tenant.database_name}, got #{current_db}"
        end
        
        # Ensure schema_migrations exists
        unless connection.table_exists?(:schema_migrations)
          puts "  → Creating schema_migrations table..."
          connection.execute(<<-SQL)
            CREATE TABLE IF NOT EXISTS schema_migrations (
              version varchar NOT NULL PRIMARY KEY
            );
          SQL
        end
        
        # Get already-run migrations
        migrated_versions = connection.select_values("SELECT version FROM schema_migrations")
        
        # Run pending migrations
        migration_files.each do |migration_file|
          version = File.basename(migration_file).split('_').first
          
          next if migrated_versions.include?(version)
          
          puts "  → Running #{File.basename(migration_file)}..."
          
          load migration_file
          class_name = File.basename(migration_file, '.rb').split('_')[1..-1].map(&:capitalize).join
          migration_class = class_name.constantize
          
          # make migration use TenantConnection, not ActiveRecord::Base
          migration_class = Class.new(migration_class) do
            def connection
              TenantConnection.connection
            end
          end

          migration_instance = migration_class.new
          migration_instance.migrate(:up)

          TenantConnection.connection.execute("INSERT INTO schema_migrations (version) VALUES ('#{version}')")
          puts "  ✓ Migrated"
          
          # migration_instance = migration_class.new
          # migration_instance.migrate(:up)
          #
          # connection.execute("INSERT INTO schema_migrations (version) VALUES ('#{version}')")
          # puts "  ✓ Migrated"
        end
        
        puts "  ✓ Success"
        migrated += 1
      rescue => e
        puts "  ✗ Failed"
        puts "  Error: #{e.class}"
        puts "  Message: #{e.message}"
        
        # Show the exact file and line where the error originated
        if e.backtrace && e.backtrace.any?
          file_line = e.backtrace.first
          puts "  Location: #{file_line}"
        end
        
        puts "  This might be the same or more helpful: "
        app_trace = e.backtrace.find { |line| line.include?(Rails.root.to_s) }
        puts "    Our Code Location: #{app_trace || e.backtrace.first}"
        
        if ENV['VERBOSE']
          puts "\n  Backtrace:"
          puts e.backtrace.first(10).map { |line| "    #{line}" }.join("\n")
        end
        failed += 1
        failed_tenants << { tenant: tenant, error: e }
      ensure
        TenantConnection.clear!
      end
    end
    
    puts "\n" + "="*60
    puts "=== Migration Summary ==="
    puts "="*60
    puts "Total tenants: #{tenants.count}"
    puts "Migrated successfully: #{migrated}"
    puts "Failed: #{failed}"
    
    if failed_tenants.any?
      puts "\nFailed Tenants:"
      failed_tenants.each do |failure|
        puts "  ✗ #{failure[:tenant].domain} (#{failure[:tenant].database_name})"
        puts "    #{failure[:error].message}"
      end
    end
  end
  
  
  
  
  
  
  
  
  # bundle exec rake tenants:count_tables_per_tenant --trace
  desc "count tables per tenant"
  task count_tables_per_tenant: :environment do 
    tenants = Tenant.where(active: true).to_a
    puts "Found #{tenants.count} active tenant(s)\n"
    
    tenants.each do |tenant|
      tenant.switch!
      count = TenantConnection.connection.tables.count
      puts "#{tenant.database_name} --- #{count}"
    end
  end
  
  desc "count migrations ran per tenant"
  task count_migrations_ran_per_tenant: :environment do 
    tenants = Tenant.where(active: true).to_a
    puts "Found #{tenants.count} active tenant(s)\n"
    
    tenants.each do |tenant|
      tenant.switch!
      count = TenantConnection.connection.execute("SELECT COUNT(*) FROM schema_migrations").first["count"]
      puts "#{tenant.database_name} --- #{count}"
    end
  end
  
  

  # bundle exec rake tenants:insert_without_migrations\[polly.pollitify,pollitify\] --trace
  desc "Create tenant without running migrations"
  task :insert_without_migrations, [:domain,:company,:database] => :environment do |t, args|
    domain = args[:domain] || 'legacy.yourdomain.com'
    company = args[:company] || domain
    database = args[:database] || ''
    
    raise "Database MUST be specified" if database.blank?
    
    puts "\n=== Creating Tenant from Legacy Database ==="
    puts "Domain: #{domain}"
    puts "Company: #{company}"
    puts "Database: #{database}"
    puts ""
    print "Continue? (y/n): "
    
    response = STDIN.gets.chomp
    exit unless response.downcase == 'y'
    
    tenant = Tenant.create!(
      domain: domain,
      database_name: database,
      company_name: company
    )
    puts "✓ Created tenant record (ID: #{tenant.id})"
  end


  
  
  
  # bundle exec rake tenants:create_from_legacy\[polly.pollitify\] --trace
  desc "Create initial tenant from legacy database"
  task :create_from_legacy, [:domain] => :environment do |t, args|
    domain = args[:domain] || 'legacy.yourdomain.com'
    
    #new_db_name = "polly_#{domain.gsub(/[^a-z0-9]/i, '_').downcase}"
    new_db_name = Tenant.get_database_name_from_host(domain)
    
    puts "\n=== Creating Tenant from Legacy Database ==="
    puts "Domain: #{domain}"
    puts "Current DB: polly2_take20aaa_#{Rails.env}"
    puts "New DB: #{new_db_name}"
    puts ""
    print "Continue? (y/n): "
    
    response = STDIN.gets.chomp
    exit unless response.downcase == 'y'
    
    tenant = Tenant.create!(
      domain: domain,
      database_name: new_db_name,
      company_name: 'Legacy Client'
    )
    puts "✓ Created tenant record (ID: #{tenant.id})"
    
    puts "\n⚠️  Close all connections to the database now!"
    puts "Press Enter when ready..."
    STDIN.gets
    
    old_db = "polly2_take20aaa_#{Rails.env}"
    
    begin
      puts "→ Terminating existing connections to #{old_db}..."
      ActiveRecord::Base.connection.execute(<<-SQL)
        SELECT pg_terminate_backend(pg_stat_activity.pid)
        FROM pg_stat_activity
        WHERE pg_stat_activity.datname = '#{old_db}'
          AND pid <> pg_backend_pid();
      SQL
      
      puts "→ Renaming database..."
      ActiveRecord::Base.connection.execute(
        "ALTER DATABASE #{old_db} RENAME TO #{new_db_name};"
      )
      
      puts "✓ Renamed database: #{old_db}#{new_db_name}"
      
      puts "\n⚠️  IMPORTANT: Update your database.yml:"
      puts "  Change 'database: #{old_db}'"
      puts "  To:     'database: #{new_db_name}'"
      
    rescue => e
      puts "✗ Failed to rename database"
      puts "Error: #{e.class} - #{e.message}"
      puts "\nYou may need to do this manually:"
      puts "  ALTER DATABASE #{old_db} RENAME TO #{new_db_name};"
      tenant.destroy
      exit 1
    end
  end
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  #bundle exec rake tenants:create\[helpfulhippie.com,HelpfulHippie\]
  desc "Create a new tenant database"
  task :create, [:domain, :company_name] => :environment do |t, args|
    domain = args[:domain]
    raise "Domain required: rails tenants:create[example.com]" unless domain
    
    puts "\n=== Creating New Tenant ==="
    puts "Domain: #{domain}"
    puts "Company: #{args[:company_name] || '(not specified)'}"
    
    tenant = Tenant.new(domain: domain, company_name: args[:company_name])
    
    if tenant.save
      puts "✓ Tenant record created"
      puts "  ID: #{tenant.id}"
      puts "  Database: #{tenant.database_name}"
      
      puts "\n→ Creating database: #{tenant.database_name}..."
      config = tenant.connection_config
      ActiveRecord::Tasks::DatabaseTasks.create(config)
      puts "✓ Database created"
      
      puts "\n→ Running migrations..."
      tenant.switch!
      
      current_db = ActiveRecord::Base.connection.current_database
      puts "✓ Connected to: #{current_db}"
      
      connection = ActiveRecord::Base.connection
      unless connection.table_exists?(:schema_migrations)
        connection.execute(<<-SQL)
          CREATE TABLE IF NOT EXISTS schema_migrations (
            version varchar NOT NULL PRIMARY KEY
          );
        SQL
      end
      
      # Run migrations manually
      migrations_path = Rails.root.join("db/migrate")
      migration_files = Dir[migrations_path.join("*.rb")].sort
      
      # migration_files.each do |migration_file|
      #   version = File.basename(migration_file).split('_').first
      #
      #   puts "→ Running #{File.basename(migration_file)}..."
      #   load migration_file
      #   class_name = File.basename(migration_file, '.rb').split('_')[1..-1].map(&:capitalize).join
      #   migration_class = class_name.constantize
      #   migration_instance = migration_class.new
      #   migration_instance.migrate(:up)
      #
      #   connection.execute("INSERT INTO schema_migrations (version) VALUES ('#{version}')")
      # end
      
      migration_files.each do |migration_file|
        version = File.basename(migration_file).split('_').first

        puts "→ Running #{File.basename(migration_file)}..."
        load migration_file

        begin
          class_name = File.basename(migration_file, '.rb')
                        .split('_')[1..-1]
                        .join('_')
                        .camelize
          migration_class = class_name.constantize
        rescue NameError
          # fallback for gem migrations with namespaces or unexpected names
          migration_class = ObjectSpace.each_object(Class).find do |klass|
            klass < ActiveRecord::Migration && klass.name.present?
          end
        end

        migration_instance = migration_class.new
        migration_instance.migrate(:up)

        connection.execute("INSERT INTO schema_migrations (version) VALUES ('#{version}')")
      end
      
      
      TenantConnection.clear!
      
      puts "\n✓ Tenant created successfully"
      puts "  Domain: #{tenant.domain}"
      puts "  Database: #{tenant.database_name}"
      puts "  Company: #{tenant.company_name}"
    else
      puts "✗ Failed to create tenant:"
      tenant.errors.full_messages.each { |msg| puts "  - #{msg}" }
      exit 1
    end
  end
  
  
  
  
  
  
  
  
  
  
  
  # bundle exec rake tenants:seed
  desc "Run custom seed for all tenants"
  task seed: :environment do
    Tenant.where(active: true).find_each do |tenant|
      puts "\n→ Seeding #{tenant.domain} (#{tenant.database_name})..."
      tenant.switch!
      
      current_db = ActiveRecord::Base.connection.current_database
      puts "  Connected to: #{current_db}"
      
      if File.exist?(Rails.root.join('db', 'seeds.rb'))
        load Rails.root.join('db', 'seeds.rb')
        puts "  ✓ Seeded"
      else
        puts "  ⚠ No seeds.rb file found"
      end
      
      TenantConnection.clear!
    end
  end
  
  
  
  
  
  
  
  
  
  
  # bundle exec rake tenants:seed_one\[helpfulhippie.com]
  desc "Run custom seed for specific tenant"
  task :seed_one, [:domain] => :environment do |t, args|
    tenant = Tenant.find_by!(domain: args[:domain])
    
    puts "\n→ Seeding #{tenant.domain} (#{tenant.database_name})..."
    tenant.switch!
    
    current_db = ActiveRecord::Base.connection.current_database
    puts "  Connected to: #{current_db}"
    
    if File.exist?(Rails.root.join('db', 'seeds.rb'))
      load Rails.root.join('db', 'seeds.rb')
      puts "✓ Seeded #{tenant.domain}"
    else
      puts "✗ No seeds.rb file found"
    end
    
    TenantConnection.clear!
  end
  
  
  
  
  
  
  
  
  
  
  
  
  # bundle exec rake tenants:list
  desc "List all tenants"
  task list: :environment do
    puts "\n=== Tenants ==="
    tenants = Tenant.order(:domain).to_a
    
    if tenants.empty?
      puts "No tenants found"
    else
      tenants.each do |tenant|
        status = tenant.active? ? "✓" : "✗"
        puts "#{status} #{tenant.domain.ljust(30)}#{tenant.database_name}"
      end
      puts "\nTotal: #{tenants.count} tenant(s)"
    end
  end
  
  
  
  
  
  
  
  
  
  
  
  # bundle exec rake tenants:drop\[helpfulhippie.com]
  desc "Drop a tenant database (DANGEROUS)"
  task :drop, [:domain] => :environment do |t, args|
    domain = args[:domain]
    raise "Domain required: rails tenants:drop[example.com]" unless domain
    
    tenant = Tenant.find_by!(domain: domain)
    
    puts "\n⚠️  WARNING: This will permanently delete the database for #{domain}"
    puts "Database: #{tenant.database_name}"
    print "Type the domain name to confirm: "
    
    confirmation = STDIN.gets.chomp
    if confirmation == domain
      puts "\n→ Dropping database..."
      config = tenant.connection_config
      ActiveRecord::Tasks::DatabaseTasks.drop(config)
      
      puts "→ Deleting tenant record..."
      tenant.destroy
      
      puts "✓ Dropped tenant database and record"
    else
      puts "✗ Confirmation failed. Aborted."
    end
  end
  
  
  
  
  
  
  
  # Helper method to get registry config
  def get_registry_config
    if Rails.env.production?
      if ENV['REGISTRY_DATABASE_URL'].present?
        {
          'adapter' => 'postgresql',
          'url' => ENV['REGISTRY_DATABASE_URL']
        }
      else
        raise "REGISTRY_DATABASE_URL environment variable not set"
      end
    else
      # Development or test
      db_configs = Rails.configuration.database_configuration
      env_config = db_configs[Rails.env]
      
      # Handle both flat and nested config structures
      registry_config = if env_config.is_a?(Hash) && env_config.key?('registry')
        env_config['registry']
      elsif env_config.is_a?(Hash) && env_config.key?(:registry)
        env_config[:registry]
      else
        # Fallback: manually build config
        {
          'adapter' => 'postgresql',
          'encoding' => 'unicode',
          'pool' => 5,
          'timeout' => 5000,
          'port' => 5432,
          'host' => 'localhost',
          'database' => "polly_registry_#{Rails.env}"
        }
      end
      
      registry_config
    end
  end
end

Running Your Tests

As I closed in on finishing this blog post, I went to re-run my tests and found that I was having migration issues that I haven’t seen in quite a long time (say Rails 3). My guess is that my low level changes towards having my own migration engine are difficult for Rails to accept so you may need to use the low level test context commands:

bin/rails db:drop RAILS_ENV=test
bin/rails db:create RAILS_ENV=test
bin/rails db:migrate RAILS_ENV=test
bin/rails db:test:prepare

I haven’t needed these in a very long time but I found that I needed them now. They are now encapsulated into bin/test_setup.

Final Steps aka Welcome to Hell aka Deploy

At the end of a long project, you just want things to work – but what you want doesn’t really matter – details matter. I went to deploy this and, naturally, it failed. And, to a limited extent, I reacted poorly. I was cognitively exhausted and I should have expected issues:

  1. Locally I rely on a postgres database with no security. Yes its a bad practice but it makes local development easy and means the dev team doesn’t have to synchronize credentials.
  2. Deploy in Rails always, always sucks monkey balls. Yes Kamal is better than previous ball sucking attempts but, ultimately, it is still deploy so …

Happily an afternoon away with some NON CODING time (I built the foundation for my upcoming greenhouse) and then an early night sleep gave me some insights:

  1. My production environment has a single database connection based on the postgres database url patttern or: DATABASE_URL=postgres://user:pass@host:5432/main_db
  2. I need at least two connection definitions:

Here is the start on the database.yml:

    production:
      primary: &primary # registry DB
        adapter: postgresql
        url: <%= ENV["REGISTRY_DATABASE_URL"] %>

      tenant:
        adapter: postgresql
        host: <%= ENV["DB_HOST"] %>
        port: <%= ENV["DB_PORT"] || 5432 %>
        username: <%= ENV["DB_USER"] %>
        password: <%= ENV["DB_PASS"] %>
        database: <%= ENV["TENANT_DATABASE_NAME"] %> # this will be overridden dynamically

The key thing to realize here is that FOR PRODUCTION, the primary database is THE REGISTRY DATABASE. This is such a different thing than development that it took laying a foundation for a (quite large) greenhouse to wrap my head around it.

At this point I did a deploy and it FAILED.

Now the thing to understand about Kamal is that Kamal:

IS HARD WIRED TO WORK THE WAY THAT 37 SIGNALS WANTS AND NEEDS

And one of the things that 37 Signals wants and needs is a health check at the of deployment. If that health check fails then the container is abandoned and the previous container is left intact. This is great for zero downtime deploys and ANNOYING AS HELL when you need code to get to a server so you can use that code. In my case:

  1. I deployed.
  2. It failed.
  3. I logged in and my database.yml changes weren’t there.

You should know that I tried mightily to get past the health check and never succeeded.

And Now We Resume Deploy…

This did, however, remind me to check server side for REGISTRY_DATABASE_URL:

echo $REGISTRY_DATABASE_URL

which gave me nothing. It did, however, remind me that I had written this crap down once before [here]https://new.fuzzyblog.io/2025/10/07/adding-an-environment-variable-to-kamal/. What I was doing was adding a new environment variable to kamal which is a 3 step process:

  1. Add it to the .env.production source file (THIS IS NEVER CHECKED INTO VERSION CONTROL BTW).
  2. Add it to .kamal/secrets
  3. Reference it in deploy.yml
  4. Deploy.

Here’s a diff of the changes I made:

❯ git diff .kamal/secrets
diff --git a/.kamal/secrets b/.kamal/secrets
index a095b1b..a1ceae9 100644
--- a/.kamal/secrets
+++ b/.kamal/secrets
@@ -35,6 +35,7 @@ KAMAL_REGISTRY_PASSWORD=$KAMAL_REGISTRY_PASSWORD
 #RAILS_MASTER_KEY=$(cat config/master.key)

 DATABASE_URL=$DATABASE_URL
+REGISTRY_DATABASE_URL=$REGISTRY_DATABASE_URL
 RAILS_SERVE_STATIC_FILES=$RAILS_SERVE_STATIC_FILES
 #MAILGUN_API_KEY=$MAILGUN_API_KEY
 HONEYBADGER_API_KEY=$HONEYBADGER_API_KEY

polly on 📙 main [📝] via 🐳 desktop-linux via  v16.3.0 via 💎 v3.4.3 on ☁️  (us-east-2)
❯ git diff config/deploy.yml
diff --git a/config/deploy.yml b/config/deploy.yml
index 34c48e8..f42d8c0 100644
--- a/config/deploy.yml
+++ b/config/deploy.yml
@@ -59,6 +59,7 @@ env:
   secret:
     - RAILS_MASTER_KEY
     - DATABASE_URL
+    - REGISTRY_DATABASE_URL
     - SECRET_KEY_BASE
     - RAILS_SERVE_STATIC_FILES
     - HONEYBADGER_API_KEY

And, so, I deployed. And it .. failed. One of the most useful engineering techniques that I know when you are in this stage of work is really, really simple – WALK AWAY FROM THE KEYBOARD. And I know that you don’t want to do that – you want nothing more than to finish and get onto the type of coding you like (in my case, I’m an app developer not a systems guy; I can do systems work but I loathe it). In this case, I got up did a literal mountain of dishes and just let my brain work the problem out. Here’s what I came up with:

  1. My first step was to check if the REGISTRY_DATABASE_URL was there in the server. And it was.
  2. No I can’t explain how I could have the REGISTRY_DATABASE_URL when my deploy had failed. Black Magic? Sacrificing a goat in the dark of the moon while a coven of naked humans danced by the fire? Honest to the FSM, I do not know. But I had it so that was a win.
  3. As I looked at the REGISTRY_DATABASE_URL and I confirmed server side that it worked both in the linux shell with ECHO $REGISTRY_DATABASE_URL and in the Rails console ENV[‘REGISTRY_DATABASE_URL’] versus the $DATABASE_URL, I noticed the some things:
    echo $REGISTRY_DATABASE_URL
    postgresql://pollitify_hub:PASSWORD@db-host:5432/polly_registry_production
    
    echo $DATABASE_URL
    $DATABASE_URL
    postgres://pollitify_hub:PASSWORD@host.docker.internal/team_pollitify_com

Two thoughts come to mind:

  1. db-host is wrong in the $REGISTRY_DATABASE_URL
  2. Is the database polly_registry_production even created yet?

So my first thing was to update .env.production and fix the REGISTRY_DATABASE_URL value with the right value for db-host. And then I ran a deploy. Please note that this .env.production isn’t in git, I didn’t even need to do the add / commit / push three step.

And now my errors went from:

ERROR Failed to boot web on polly.pollitify.com
  INFO First web container is unhealthy on polly.pollitify.com, not booting any other roles
  INFO [6b38bbf5] Running docker container ls --all --filter name=^polly-web-8015cd22178d0a293d49450a7979d76f58e10c7f$ --quiet | xargs docker logs --timestamps 2>&1 on polly.pollitify.com
  INFO [6b38bbf5] Finished in 0.255 seconds with exit status 0 (successful).
 ERROR 2025-11-14T08:52:30.785471637Z bin/rails aborted!
2025-11-14T08:52:30.785537931Z ActiveRecord::DatabaseConnectionError: There is an issue connecting with your hostname: db-host. (ActiveRecord::DatabaseConnectionError)
2025-11-14T08:52:30.785546417Z
2025-11-14T08:52:30.785551707Z Please check your database configuration and ensure there is a valid connection to your database.
2025-11-14T08:52:30.785556726Z
2025-11-14T08:52:30.785561154Z
2025-11-14T08:52:30.785565534Z Caused by:
2025-11-14T08:52:30.785570142Z PG::ConnectionBad: could not translate host name "db-host" to address: Temporary failure in name resolution (PG::ConnectionBad)
2025-11-14T08:52:30.785763494Z
2025-11-14T08:52:30.788363353Z Tasks: TOP => db:prepare
2025-11-14T08:52:30.788405451Z (See full trace by running task with --trace)
2025-11-14T08:52:30.915196749Z ** [Honeybadger] Reporting error id=e9792df3-80f2-4a9f-adbd-6eaaf003c1dd level=1 pid=8
2025-11-14T08:52:30.915224792Z ** [Honeybadger] Success ⚡ https://app.honeybadger.io/notice/e9792df3-80f2-4a9f-adbd-6eaaf003c1dd id=e9792df3-80f2-4a9f-adbd-6eaaf003c1dd code=201 level=1 pid=8

to

 ERROR Failed to boot web on polly.pollitify.com
  INFO First web container is unhealthy on polly.pollitify.com, not booting any other roles
  INFO [3b596fa7] Running docker container ls --all --filter name=^polly-web-8015cd22178d0a293d49450a7979d76f58e10c7f$ --quiet | xargs docker logs --timestamps 2>&1 on polly.pollitify.com
  INFO [3b596fa7] Finished in 0.281 seconds with exit status 0 (successful).
 ERROR 2025-11-14T09:34:34.455461852Z bin/rails aborted!
2025-11-14T09:34:34.455492790Z ActiveRecord::StatementInvalid: PG::InsufficientPrivilege: ERROR:  permission denied for schema public (ActiveRecord::StatementInvalid)
2025-11-14T09:34:34.455497198Z LINE 1: CREATE TABLE "action_text_rich_texts" ("id" bigserial primar...
2025-11-14T09:34:34.455500625Z                      ^
2025-11-14T09:34:34.455503230Z /rails/db/schema.rb:17:in `block in <main>'
2025-11-14T09:34:34.455506676Z /rails/db/schema.rb:13:in `<main>'
2025-11-14T09:34:34.455509532Z    

Yes it is still an error but that’s a world of difference. Still the first thing is to validate that the url is now what it should be and, yes, there are both a DATABASE_URL and a REGISTRY_DATABASE_URL.

I next ran a:

bundle exec rake db:create

Which told me:

bundle exec rake db:create
Database 'team_pollitify_com' already exists

So this confirmed that I really hadn’t flipped over my main database from the traditional structure to the tenant structure (where Registry becomes the main database). And I then realized that this has grave migration implications. My additional concern was that the polly_registry_production database wasn’t there so I ended up exiting the ssh into the container and going into the underlying hetzner box and using a native psql prompt and then doing a \l

 \l
                                                                  List of databases
           Name            |     Owner     | Encoding | Locale Provider |   Collate   
---------------------------+---------------+----------+-----------------+-------------
 pollitify_hub_production  | pollitify_hub | UTF8     | libc            | en_US.UTF-8 
 polly_pollitify_com       | pollitify_hub | UTF8     | libc            | en_US.UTF-8 
 polly_registry_production | postgres      | UTF8     | libc            | en_US.UTF-8 
 postgres                  | postgres      | UTF8     | libc            | en_US.UTF-8 
 team_pollitify_com        | pollitify_hub | UTF8     | libc            | en_US.UTF-8 
 template0                 | postgres      | UTF8     | libc            | en_US.UTF-8 
                           |               |          |                 |             
 template1                 | postgres      | UTF8     | libc            | en_US.UTF-8 
                           |               |          |                 |             

Interlude: Stepping Back and Re-Assessing

At this point I started to realize that I had actually, perhaps, made a terrible series of mistakes. This was when I put down the keyboard and went outside to work on my greenhouse and think about my approach.

What I came back with was the following:

  1. I should have done this systems level work on DAY 1 of the project. Putting it off made until launch made me happy as an application developer (I could write the code I liked) but it meant that the bigger the project got, the harder it would be to make these changes. Sigh.
  2. Rails is fundamentally aligned towards having a primary database. For myself, that primary database is the application code. And then there’s the registry database. And then there are the tenant databases which are the same as the primary database. Except in production the primary database needs to be the registry database. Got that? Really? I’m not sure I do.
  3. For a multitenant application, the primary database – EVEN IN DEVELOPMENT – is the registry. The reason for this is migrations. Right now I have a confusing structure where my primary db is different between production and development and that’s always going to be weird.
  4. This is when I realized that what I wanted – which didn’t exist – was a Rails 8 starter application for multitenancy which was setup this way (and maybe I’ll try and extract that from my application when I’m done).

It isn’t surprising that migrations came up as a key issue – migrations are part of persistent storage and scaling persistent storage is always one of the hardest parts of everything.

Now this pulled me into thinking about how should a multi tenant application really work from a domain perspective (let’s assume that the base domain is foo.com):

  1. Go to foo.com and it is marketing and sign up.
  2. Go to customername.foo.com and it is their data.

My situation is a little bit different since I have an entirely different Rails codebase running on my base domain but the same principle can be applied:

  1. Go to signup.foo.com and you get the sign up process. This will add people to the customers table.
  2. (Implementation Detail) After a custom signs up and is approved, there is an admin tool which creates the entry in the registry database and sends the customer an email granting them access.
  3. Go to customername.foo.com and it is their data.

Since I am already doing database connection logic based on the host, I can simply (chuckle) connect to the primary database when it is signup.foo.com and then for everything else I look up in the tenant database -or- if the domain is wrong, branch them back within signup to an error page.

The key thing here is going to be messing about with the structure of db directory:

 tree -d db
db
├── migrate
├── registry_migrate
└── seed

Basically the contents of db need to change as follows:

  1. Move the contents of migrate to tenant_migrate.
  2. Move the contents of registry_migrate over to migrate.
  3. Rejigger all logic to make the registry the main database.

This is the place where even though I’ve merged all my code back into main, I now need to branch again and work in that branch because this might not work. I know I can make my current situation work if I grind at it hard enough. But this new approach should actually be quite a bit easier to implement.

From an engineering perspective, I refer to this as Problem Solving by Inversion. For whatever reason, I have often found that when your coding gets harder and harder and you keep thinkings “this time”, “this time”, “this time”, each time hitting your face against a solid surface (much like the bee in Bee Movie):

this time

then maybe you need to try inverting – your code, your assumptions, etc.

Post Interlude 1: Restructuring Migrations

The first step is to restructure the db directory. A few mkdirs, git mv commands and you get:

tree -d db
db
├── migrate
├── seed
└── tenant_migrate

Post Interlude 2: Thinking About Icky Business Stuff

One of the things I’ve said for years is that when you use a framework and you fight its core nature, you are setting yourself up for a world of hurt. That’s why this single blog post is almost 7,000 words so far. Rails is oriented in two primary ways:

  1. A single primary data store defined in database.yml. Yes you can now have separate data stores as defined in database.yml by named keys but the Rails 8 api for this is … weird. Of all the above code, I had more issues with a separate “registry” database than I did anything else. Honestly it was gross and just kept getting more and more convoluted. That’s a code smell that says you are traveling the wrong path.
  2. The Majestic Monolith. That’s the term for a single giant Rails code base that does everything. It was how DHH first built basecamp and it is the model that Rails is good at.

Thinking about all this makes me realize that my Majestic Monolith is:

  1. My current application code BUT with storage of USER data in individual postgres databases
  2. A sign up process (something I’ve actually not built yet)
  3. Database models and tables like users, billing_plans, user_billing_plans.
  4. Promotional web pages to run the signup process, document the product, etc.
  5. A dedicated domain where this can live. Prior to this point, there actually hasn’t been a dedicated website for our product, just stuff hosted off our community site.

The key thing tho is that items 1 to 5 above ARE ALL PART OF THE MAJESTIC MONOLITH. No separate code base for sales, no separate site for sign up, no cross site apis, etc. All of this is just ONE rails code base, one monolith.

Post Interlude 3: Rethinking Models

This brings us to the task of thinking about models. Above we had tenant which defined the tenant database as well as tenant_connection (which provided the required abstract class). But tenant both defined that database name as well as handled host and connection details. A cleaner path is: