Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
d00bea8
fix: add DateTime/Decimal parameter encoding for Oban compatibility
ocean Jan 12, 2026
57c5846
fix: encode plain maps to JSON before passing to NIF
ocean Jan 13, 2026
310029a
fix: add catch-all clause to column_default/1 for robustness
ocean Jan 13, 2026
7671d65
feat: Add comprehensive type encoding support and tests
ocean Jan 13, 2026
3f1d114
docs: Add type encoding implementation summary
ocean Jan 13, 2026
3d94b37
docs: Add guidance for relative dates and third-party date types
ocean Jan 13, 2026
4682df7
chore: Remove TYPE_ENCODING_SUMMARY.md and .beads files from tracking
ocean Jan 13, 2026
b6aa213
chore: Beads config
ocean Jan 13, 2026
2e18459
chore: Beads config
ocean Jan 13, 2026
526e5bf
fix: add automatic map encoding and improve parameter encoding tests
ocean Jan 13, 2026
aef0408
test: update plain maps encoding test to use raw map parameter
ocean Jan 13, 2026
b62a947
feat: add warning logging for unsupported default value types in migr…
ocean Jan 13, 2026
e21360f
test: consolidate type encoding tests by merging investigation into i…
ocean Jan 13, 2026
5757a16
Fix boolean encoding tests to expect correct SQLite representation
ocean Jan 13, 2026
55990b7
chore: Remove Oban mentions, as these types could come from anywhere
ocean Jan 13, 2026
e836cbc
Improve error handling for map parameter JSON encoding
ocean Jan 13, 2026
f393b70
Remove unused Ecto schema modules from type encoding tests
ocean Jan 13, 2026
0438907
Add comprehensive type encoding tests for float, NULL edge cases, and…
ocean Jan 13, 2026
03da1b2
chore: Update beads issue tracking after test review
ocean Jan 13, 2026
2c99e60
test: strengthen type encoding assertions for better precision
ocean Jan 13, 2026
b4d6dff
Fix test precision issues in type_encoding_implementation_test
ocean Jan 13, 2026
634efe2
chore: Fix formatting
ocean Jan 13, 2026
7d47fd8
test: tighten assertions in type_encoding_implementation_test.exs for…
ocean Jan 13, 2026
bf0bb24
refactor: fix credo warnings and refactoring opportunities
ocean Jan 13, 2026
0ffd576
test: tighten float comparison assertion for deterministic count
ocean Jan 13, 2026
7773886
refactor: address code quality feedback for type_encoding_implementat…
ocean Jan 13, 2026
87bfaa9
test: make type affinity assertion deterministic
ocean Jan 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .beads/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,11 @@ beads.left.meta.json
beads.right.jsonl
beads.right.meta.json

# Sync state (local-only, per-machine)
# These files are machine-specific and should not be shared across clones
.sync.lock
sync_base.jsonl

# NOTE: Do NOT add negation patterns (e.g., !issues.jsonl) here.
# They would override fork protection in .git/info/exclude, allowing
# contributors to accidentally commit upstream issue databases.
Expand Down
2 changes: 1 addition & 1 deletion .beads/last-touched
Original file line number Diff line number Diff line change
@@ -1 +1 @@
el-6r5
el-1p2
6 changes: 5 additions & 1 deletion .claude/settings.local.json
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,11 @@
"Bash(git commit:*)",
"Bash(git push)",
"Bash(git --no-pager status)",
"Bash(cargo deny check:*)"
"Bash(cargo deny check:*)",
"Bash(gh pr diff:*)",
"Bash(gh pr checks:*)",
"Bash(gh run view:*)",
"Bash(gh pr checkout:*)"
],
"deny": [],
"ask": []
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -41,5 +41,7 @@ z_ecto_libsql_test*

# bv (beads viewer) local config and caches
.bv/

# Implementation summaries and temporary docs
TEST_AUDIT_REPORT.md
TEST_COVERAGE_ISSUES_CREATED.md
193 changes: 193 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2610,6 +2610,199 @@ export TURSO_AUTH_TOKEN="eyJ..."
- 🌍 **Global distribution** via Turso edge
- 💪 **Offline capability** - works without network

### Type Encoding and Parameter Conversion

EctoLibSql automatically converts Elixir types to SQLite-compatible formats. Understanding these conversions is important for correct database usage.

#### Automatically Encoded Types

The following types are automatically converted when passed as query parameters:

##### Temporal Types

```elixir
# DateTime → ISO8601 string
dt = DateTime.utc_now()
SQL.query!(Repo, "INSERT INTO events (created_at) VALUES (?)", [dt])
# Stored as: "2026-01-13T03:45:23.123456Z"

# NaiveDateTime → ISO8601 string
dt = NaiveDateTime.utc_now()
SQL.query!(Repo, "INSERT INTO events (created_at) VALUES (?)", [dt])
# Stored as: "2026-01-13T03:45:23.123456"

# Date → ISO8601 string
date = Date.utc_today()
SQL.query!(Repo, "INSERT INTO events (event_date) VALUES (?)", [date])
# Stored as: "2026-01-13"

# Time → ISO8601 string
time = Time.new!(14, 30, 45)
SQL.query!(Repo, "INSERT INTO events (event_time) VALUES (?)", [time])
# Stored as: "14:30:45.000000"

# Relative dates (compute absolute date first, then pass)
tomorrow = Date.add(Date.utc_today(), 1) # Becomes a Date struct
SQL.query!(Repo, "INSERT INTO events (event_date) VALUES (?)", [tomorrow])

# Third-party date types (Timex, etc.) - pre-convert to standard types
# ❌ NOT SUPPORTED: Timex.DateTime or custom structs
# ✅ DO THIS: Convert to native DateTime first
timex_dt = Timex.now()
native_dt = Timex.to_datetime(timex_dt) # Convert to DateTime
SQL.query!(Repo, "INSERT INTO events (created_at) VALUES (?)", [native_dt])
```

##### Boolean Values

```elixir
# true → 1, false → 0
# SQLite uses integers for booleans
SQL.query!(Repo, "INSERT INTO users (active) VALUES (?)", [true])
# Stored as: 1

SQL.query!(Repo, "INSERT INTO users (active) VALUES (?)", [false])
# Stored as: 0

# Works with WHERE clauses
SQL.query!(Repo, "SELECT * FROM users WHERE active = ?", [true])
# Matches rows where active = 1
```

##### Decimal Values

```elixir
# Decimal → string representation
decimal = Decimal.new("123.45")
SQL.query!(Repo, "INSERT INTO prices (amount) VALUES (?)", [decimal])
# Stored as: "123.45"
```

##### NULL/nil Values

```elixir
# nil → NULL
SQL.query!(Repo, "INSERT INTO users (bio) VALUES (?)", [nil])
# Stored as SQL NULL

# :null atom → nil → NULL (v0.8.3+)
# Alternative way to represent NULL
SQL.query!(Repo, "INSERT INTO users (bio) VALUES (?)", [:null])
# Also stored as SQL NULL

# Both work identically:
SQL.query!(Repo, "SELECT * FROM users WHERE bio IS NULL") # Matches both
```

##### UUID Values

```elixir
# Ecto.UUID strings work directly (already binary strings)
uuid = Ecto.UUID.generate()
SQL.query!(Repo, "INSERT INTO users (id) VALUES (?)", [uuid])
# Stored as: "550e8400-e29b-41d4-a716-446655440000"

# Works with WHERE clauses
SQL.query!(Repo, "SELECT * FROM users WHERE id = ?", [uuid])
```

#### Type Encoding Examples

```elixir
defmodule MyApp.Examples do
def example_with_multiple_types do
import Ecto.Adapters.SQL

now = DateTime.utc_now()
user_active = true
amount = Decimal.new("99.99")

# All types are automatically encoded
query!(Repo,
"INSERT INTO transactions (created_at, active, amount) VALUES (?, ?, ?)",
[now, user_active, amount]
)
end

def example_with_ecto_queries do
import Ecto.Query

from(u in User,
where: u.active == ^true, # Boolean encoded to 1
where: u.created_at > ^DateTime.utc_now() # DateTime encoded to ISO8601
)
|> Repo.all()
end

def example_with_null do
# Both are equivalent:
SQL.query!(Repo, "INSERT INTO users (bio) VALUES (?)", [nil])
SQL.query!(Repo, "INSERT INTO users (bio) VALUES (?)", [:null])

# Query for NULL values
SQL.query!(Repo, "SELECT * FROM users WHERE bio IS NULL")
end
end
```

#### Limitations: Nested Structures with Temporal Types

Nested structures (maps/lists) containing temporal types are **not automatically encoded**. Only top-level parameters are encoded.

```elixir
# ❌ DOESN'T WORK - Nested DateTime not encoded
nested = %{
"created_at" => DateTime.utc_now(), # ← Not auto-encoded
"data" => "value"
}
SQL.query!(Repo, "INSERT INTO events (metadata) VALUES (?)", [nested])
# Error: DateTime struct cannot be serialized to JSON

# ✅ WORKS - Pre-encode nested values
nested = %{
"created_at" => DateTime.utc_now() |> DateTime.to_iso8601(),
"data" => "value"
}
json = Jason.encode!(nested)
SQL.query!(Repo, "INSERT INTO events (metadata) VALUES (?)", [json])

# ✅ WORKS - Encode before creating map
dt = DateTime.utc_now() |> DateTime.to_iso8601()
nested = %{"created_at" => dt, "data" => "value"}
json = Jason.encode!(nested)
SQL.query!(Repo, "INSERT INTO events (metadata) VALUES (?)", [json])
```

**Workaround:**
When working with maps/lists containing temporal types, manually convert them to JSON strings before passing to queries:

```elixir
defmodule MyApp.JsonHelpers do
def safe_json_encode(map) when is_map(map) do
map
|> Enum.map(fn
{k, %DateTime{} = v} -> {k, DateTime.to_iso8601(v)}
{k, %NaiveDateTime{} = v} -> {k, NaiveDateTime.to_iso8601(v)}
{k, %Date{} = v} -> {k, Date.to_iso8601(v)}
{k, %Decimal{} = v} -> {k, Decimal.to_string(v)}
{k, v} -> {k, v}
end)
|> Enum.into(%{})
|> Jason.encode!()
end
end

# Usage:
nested = %{
"created_at" => DateTime.utc_now(),
"data" => "value"
}
json = MyApp.JsonHelpers.safe_json_encode(nested)
SQL.query!(Repo, "INSERT INTO events (metadata) VALUES (?)", [json])
```

---

### Limitations and Known Issues

#### freeze_replica/1 - NOT SUPPORTED
Expand Down
13 changes: 13 additions & 0 deletions lib/ecto/adapters/libsql/connection.ex
Original file line number Diff line number Diff line change
Expand Up @@ -422,6 +422,19 @@ defmodule Ecto.Adapters.LibSql.Connection do
defp column_default(value) when is_binary(value), do: " DEFAULT '#{escape_string(value)}'"
defp column_default(value) when is_number(value), do: " DEFAULT #{value}"
defp column_default({:fragment, expr}), do: " DEFAULT #{expr}"
# Handle any other unexpected types (e.g., empty maps or third-party migrations)
# Logs a warning to help with debugging while gracefully falling back to no DEFAULT clause
defp column_default(unexpected) do
require Logger

Logger.warning(
"Unsupported default value type in migration: #{inspect(unexpected)} - " <>
"no DEFAULT clause will be generated. This can occur with some generated migrations " <>
"or other third-party integrations that provide unexpected default types."
)

""
end

defp table_options(table, columns) do
# Validate mutually exclusive options (per libSQL specification)
Expand Down
30 changes: 27 additions & 3 deletions lib/ecto_libsql/native.ex
Original file line number Diff line number Diff line change
Expand Up @@ -684,7 +684,10 @@ defmodule EctoLibSql.Native do

@doc false
defp do_query(conn_id, mode, syncx, statement, args_for_execution, query, state) do
case query_args(conn_id, mode, syncx, statement, args_for_execution) do
# Encode parameters to handle complex Elixir types (maps, etc.).
encoded_args = encode_parameters(args_for_execution)

case query_args(conn_id, mode, syncx, statement, encoded_args) do
%{
"columns" => columns,
"rows" => rows,
Expand Down Expand Up @@ -749,6 +752,9 @@ defmodule EctoLibSql.Native do

@doc false
defp do_execute_with_trx(conn_id, trx_id, statement, args_for_execution, query, state) do
# Encode parameters to handle complex Elixir types (maps, etc.).
encoded_args = encode_parameters(args_for_execution)

# Detect the command type to route correctly.
command = detect_command(statement)

Expand All @@ -761,7 +767,7 @@ defmodule EctoLibSql.Native do

if should_query do
# Use query_with_trx_args for SELECT or statements with RETURNING.
case query_with_trx_args(trx_id, conn_id, statement, args_for_execution) do
case query_with_trx_args(trx_id, conn_id, statement, encoded_args) do
%{
"columns" => columns,
"rows" => rows,
Expand Down Expand Up @@ -790,7 +796,7 @@ defmodule EctoLibSql.Native do
end
else
# Use execute_with_transaction for INSERT/UPDATE/DELETE without RETURNING
case execute_with_transaction(trx_id, conn_id, statement, args_for_execution) do
case execute_with_transaction(trx_id, conn_id, statement, encoded_args) do
num_rows when is_integer(num_rows) ->
result = %EctoLibSql.Result{
command: command,
Expand Down Expand Up @@ -2167,4 +2173,22 @@ defmodule EctoLibSql.Native do
def freeze_replica(_state) do
{:error, :unsupported}
end

# Encode parameters to handle complex Elixir types before passing to NIF.
# The Rust NIF cannot serialize plain Elixir maps, so we convert them to JSON strings.
@doc false
defp encode_parameters(args) when is_list(args) do
Enum.map(args, &encode_param/1)
end

defp encode_parameters(args), do: args

@doc false
# Only encode plain maps (not structs) to JSON.
# Structs like DateTime, Decimal etc are handled in query.ex encode.
defp encode_param(value) when is_map(value) and not is_struct(value) do
Jason.encode!(value)
end

defp encode_param(value), do: value
end
55 changes: 55 additions & 0 deletions lib/ecto_libsql/query.ex
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,63 @@ defmodule EctoLibSql.Query do

def describe(query, _opts), do: query

# Convert Elixir types to SQLite-compatible values before sending to NIF.
# Rustler cannot automatically serialise complex Elixir structs like DateTime,
# so we convert them to ISO8601 strings that SQLite can handle.
#
# Supported type conversions:
# - DateTime/NaiveDateTime/Date/Time → ISO8601 strings
# - Decimal → string representation
# - true/false → 1/0 (SQLite uses integers for booleans)
# - UUID binary → string representation (if needed)
# - :null atom → nil (SQL NULL)
def encode(_query, params, _opts) when is_list(params) do
Enum.map(params, &encode_param/1)
end

def encode(_query, params, _opts), do: params

# Temporal types
defp encode_param(%DateTime{} = dt), do: DateTime.to_iso8601(dt)
defp encode_param(%NaiveDateTime{} = dt), do: NaiveDateTime.to_iso8601(dt)
defp encode_param(%Date{} = d), do: Date.to_iso8601(d)
defp encode_param(%Time{} = t), do: Time.to_iso8601(t)

# Decimal
defp encode_param(%Decimal{} = d), do: Decimal.to_string(d)

# Boolean conversion: SQLite uses 0/1 for boolean values
# This is important for queries like: where u.active == ^true
defp encode_param(true), do: 1
defp encode_param(false), do: 0

# NULL atom conversion: :null → nil (SQL NULL)
# This allows using :null in Ecto queries as an alternative to nil
defp encode_param(:null), do: nil

# Map encoding: plain maps (not structs) are encoded to JSON
# Maps must contain only JSON-serializable values (strings, numbers, booleans,
# nil, lists, and nested maps). PIDs, functions, references, and other special
# Elixir types are not serializable and will raise a descriptive error.
defp encode_param(value) when is_map(value) and not is_struct(value) do
case Jason.encode(value) do
{:ok, json} ->
json

{:error, %Jason.EncodeError{message: msg}} ->
raise ArgumentError,
message:
"Cannot encode map parameter to JSON. Map contains non-JSON-serializable value. " <>
"Maps can only contain strings, numbers, booleans, nil, lists, and nested maps. " <>
"Reason: #{msg}. Map: #{inspect(value)}"
end
end

# Pass through all other values unchanged
defp encode_param(value), do: value

# Pass through results from Native.ex unchanged.
# Native.ex already handles proper normalisation of columns and rows.
def decode(_query, result, _opts), do: result
end

Expand Down
Loading