Update to support Ecto 2.2. (#193)

* Drop support for Elixir 1.3.x, as was done in corresponding Ecto master branch during that time.

* Update deps to match Ecto 2.2 branch.

* Drop support for OTP 18.0. Continue to test against OTP 18.2 as does Ecto 2.2.x release.

* Sadly, we now have to punt on Decimal support. There's really no way to store precision accurately.

* Prepare version 2.2.0 release.

* Add note about Ecto 2.x version compatibility.

This version only works with Ecto 2.2.x. I tried it with the latest 2.1.x release and it was broken (relying on APIs that don't exist in that series).

Author: scouten

.travis.yml

@@ -1,25 +1,20 @@
 language: elixir
 
 elixir:
-  - 1.3.4
   - 1.4.4
   - 1.5.1
 
 otp_release:
-  - 18.0
+  - 18.2
   - 19.3
   - 20.0
 
 matrix:
   exclude:
-  - elixir: 1.3.4
-    otp_release: 19.3
-  - elixir: 1.3.4
-    otp_release: 20.0
   - elixir: 1.4.4
     otp_release: 20.0
   - elixir: 1.5.1
-    otp_release: 18.0
+    otp_release: 18.2
 
 script:
   - mix deps.get && ./integration/hack_out_incompatible_tests.sh && mix test --cover
@@ -28,4 +23,3 @@ script:
 branches:
   only:
     - master
-    - ecto-2.2

CHANGELOG.md

@@ -1,104 +1,16 @@
-# Changelog for v2.0
+# Changelog for v2.2.x Series
 
-This is a major rewrite of the previously-existing [`sqlite_ecto`](https://github.com/jazzyb/sqlite_ecto) that adds support for Ecto 2.1+.
-
-
-## v2.0.3
+## v2.2.0
 
 _28 August 2017_
 
-* Add sqlitex to applications in mix.exs. (Thank you, @obmarg!)
-
-
-## v2.0.2
-
-_12 August 2017_
-
-* Update for Elixir 1.5.1. (Stop using a now-deprecated API; otherwise no significant changes.)
-
-
-## v2.0.1
-
-_7 August 2017_
-
-* Formally announce availability.
-* Fix a few README and CHANGELOG issues.
-
-
-## v2.0.0
-
-_6 August 2017_
-
-* Fix a few documentation errors.
-
-
-## v2.0.0-dev.8
-
-_11 June 2017_
-
-* Improve README content a bit.
-* Make db_connection a hard dependency. (Fixes #174.)
-* Update references to several libraries.
-
-
-## v2.0.0-dev.7
-
-_06 May 2017_
-
-* Improve test coverage (now 98%).
-* Add Dogma to CI build infrastructure to help enforce code format consistency.
-* Remove `Sqlite.DbConnection` module in favor of directly calling `DbConnection` itself.
-
-
-## v2.0.0-dev.6
-
-_03 May 2017_
-
-* Fix error messages that referred to PostgreSQL instead of SQLite.
-* Improve test coverage (now 91%).
-* Fix shell script issues flagged by Ebert.
-* Clean up and simplify implementation of `Sqlite.DbConnection.Protocol`.
-
-
-## v2.0.0-dev.5
-
-_30 April 2017_
-
-* Use iodata lists as much as possible during query generation.
-* Switch to numeric placeholders across the board.
-* Bring sqlite_ecto_test.exs up to date with corresponding postgres_test.exs from Ecto.
-
-
-## v2.0.0-dev.4
-
-_17 April 2017_
-
-* Port documentation from v1 `sqlite_ecto` repo to this repo. (Thank you, @taufiqkh!)
-* Enable automated code review by Ebert. Fix some of the issues it flagged.
-
-
-## v2.0.0-dev.3
-
-_11 April 2017_
-
-* Requires sqlitex version 1.3.2 which includes an important bug fix (https://github.com/mmmries/sqlitex/pull/59).
-
-
-## v2.0.0-dev.2
-
-_26 March 2017_
-
-* **BREAKING CHANGE:** Use the name `Sqlite.Ecto2` consistently in API names. Discontinue use of name `Sqlite.Ecto` (without the `2`)
-* Ensure db_connection app is started before relying on it.
-
-
-## v2.0.0-dev.1
-
-_21 March 2017_
-
-Initial public release of version 2.0 (alpha quality).
+* Update to support Ecto 2.2.x. (Aligned version numbers to match Ecto versioning series.)
+* Drop support for Elixir 1.3.x. New minimum version is 1.4.0.
+* Drop support for OTP 18.0. New minimum version is 18.2.
+* Decimal value support, while still present in the library, is no longer officially supported.
 
 
 ## Previous versions
 
-* See the CHANGELOG.md [for the v1.x series](https://github.com/jazzyb/sqlite_ecto/blob/master/CHANGELOG.md)
+* [v2.0.x series](https://github.com/scouten/sqlite_ecto2/blob/v2.0/CHANGELOG.md)
+* [v1.x series](https://github.com/jazzyb/sqlite_ecto/blob/master/CHANGELOG.md)

README.md

@@ -5,17 +5,23 @@
 
 # sqlite_ecto2
 
-`sqlite_ecto2` is an Ecto 2.x adapter that allows you to create and maintain SQLite3 databases.
+`sqlite_ecto2` is an Ecto 2.2.x adapter that allows you to create and maintain SQLite3 databases.
 
 Read [the tutorial](./docs/tutorial.md) for a detailed example of how to setup and use a SQLite repo with Ecto, or just check-out the CliffsNotes in the sections below if you want to get started quickly.
 
 
+## Ecto Version Compatibility
+
+**IMPORTANT:** This release will _only_ work with Ecto 2.2.x. If you need compatibility with older versions of Ecto, please see:
+
+* Ecto 2.1.x -> [`sqlite_ecto2` 2.0.x series](https://github.com/scouten/sqlite_ecto2/tree/v2.0)
+* Ecto 1.x -> [`sqlite_ecto` v1.x series](https://github.com/jazzyb/sqlite_ecto)
+
+
 ## When to Use `sqlite_ecto2`
 
 *(and when not to use it ...)*
 
-If, for some reason, you still need to use Ecto 1.x, please look at [sqlite_ecto](https://github.com/jazzyb/sqlite_ecto), on which this project is based.
-
 I strongly recommend reading [Appropriate Uses for SQLite](https://sqlite.org/whentouse.html) on the SQLite site itself. All of the considerations mentioned there apply to this library as well.
 
 I will add one more: If there is *any* potential that more than one server node will need to write directly to the database at once (as often happens when using Elixir in a clustered environment), **do not use** `sqlite_ecto2`. Remember that there is no separate database process in this configuration, so each of your cluster nodes would be writing to its **own** copy of the database without any synchronization. You probably don't want that. Look for a true client/server database (Postgres, MySQL, or similar) in that case. SQLite's sweet spot is single-machine deployments (embedded, desktop, etc.).

integration/hack_out_incompatible_tests.sh

@@ -19,7 +19,11 @@ sed -i "" '/test "Repo.insert_all escape/ i\
   @tag :insert_cell_wise_defaults
 ' deps/ecto/integration_test/sql/sql.exs
 
-sed -i "" '/subqueries with select expression/ i\ 
+sed -i "" '/subqueries with map and select expression/ i\ 
+  @tag :map_boolean_in_subquery
+' deps/ecto/integration_test/sql/subquery.exs
+
+sed -i "" '/subqueries with map update and select expression/ i\ 
   @tag :map_boolean_in_subquery
 ' deps/ecto/integration_test/sql/subquery.exs
 
@@ -31,6 +35,8 @@ sed -i '/failing child foreign key/ i @tag :foreign_key_constraint' deps/ecto/in
 
 sed -i '/test "Repo.insert_all escape/ i @tag :insert_cell_wise_defaults' deps/ecto/integration_test/sql/sql.exs
 
-sed -i '/subqueries with select expression/ i @tag :map_boolean_in_subquery' deps/ecto/integration_test/sql/subquery.exs
+sed -i '/subqueries with map and select expression/ i @tag :map_boolean_in_subquery' deps/ecto/integration_test/sql/subquery.exs
+
+sed -i '/subqueries with map update and select expression/ i @tag :map_boolean_in_subquery' deps/ecto/integration_test/sql/subquery.exs
 
 fi

integration/sqlite/test_helper.exs

@@ -20,7 +20,8 @@ ExUnit.start exclude: [:array_type,
                        :map_boolean_in_subquery,
                        :upsert_all,
                        :with_conflict_target,
-                       :without_conflict_target]
+                       :without_conflict_target,
+                       :decimal_type]
 
 # Configure Ecto for support and tests
 Application.put_env(:ecto, :primary_key_type, :id)

lib/sqlite_ecto.ex

@@ -51,6 +51,7 @@ defmodule Sqlite.Ecto2 do
     do: [&json_decode/1, &Ecto.Adapters.SQL.load_embed(type, &1)]
   def loaders(:map, type), do: [&json_decode/1, type]
   def loaders({:map, _}, type), do: [&json_decode/1, type]
+  def loaders(:float, type), do: [&float_decode/1, type]
   def loaders(_primitive, type), do: [type]
 
   defp bool_decode(0), do: {:ok, false}
@@ -81,6 +82,9 @@ defmodule Sqlite.Ecto2 do
   defp json_decode(x),
     do: {:ok, x}
 
+  defp float_decode(x) when is_integer(x), do: {:ok, x / 1}
+  defp float_decode(x), do: {:ok, x}
+
   def dumpers(:binary, type), do: [type, &blob_encode/1]
   def dumpers(:binary_id, type), do: [type, Ecto.UUID]
   def dumpers(:boolean, type), do: [type, &bool_encode/1]

lib/sqlite_ecto/connection.ex

@@ -65,7 +65,7 @@ if Code.ensure_loaded?(Sqlitex.Server) do
           {:ok, value} = Ecto.DataType.dump(data_type)
           value
         %{} = value ->
-          json_library().encode!(value)
+          Ecto.Adapter.json_library().encode!(value)
         value ->
           value
       end
@@ -93,7 +93,7 @@ if Code.ensure_loaded?(Sqlitex.Server) do
       limit = limit(query, sources)
       offset = offset(query, sources)
 
-      IO.iodata_to_binary([select, from, join, where, group_by, having, order_by, limit, offset])
+      [select, from, join, where, group_by, having, order_by, limit, offset]
     end
 
     def update_all(%Ecto.Query{joins: [_ | _]}) do
@@ -107,7 +107,7 @@ if Code.ensure_loaded?(Sqlitex.Server) do
       fields = update_fields(query, sources)
       where = where(%{query | wheres: query.wheres}, sources)
 
-      IO.iodata_to_binary([prefix, fields, where | returning(query, sources, :update)])
+      [prefix, fields, where | returning(query, sources, :update)]
     end
 
     def delete_all(%Ecto.Query{joins: [_ | _]}) do
@@ -119,7 +119,7 @@ if Code.ensure_loaded?(Sqlitex.Server) do
 
       where = where(%{query | wheres: query.wheres}, sources)
 
-      IO.iodata_to_binary(["DELETE FROM ", from, where | returning(query, sources, :delete)])
+      ["DELETE FROM ", from, where | returning(query, sources, :delete)]
     end
 
     def insert(prefix, table, header, rows, on_conflict, returning) do
@@ -136,8 +136,7 @@ if Code.ensure_loaded?(Sqlitex.Server) do
         _ -> raise ArgumentError, "Upsert in SQLite must use on_conflict: :nothing"
       end
       returning = returning_clause(prefix, table, returning, "INSERT")
-      IO.iodata_to_binary(["INSERT", on_conflict, " INTO ", quote_table(prefix, table),
-                           values, returning])
+      ["INSERT", on_conflict, " INTO ", quote_table(prefix, table), values, returning]
     end
 
     defp insert_all(rows, counter) do
@@ -168,17 +167,17 @@ if Code.ensure_loaded?(Sqlitex.Server) do
 
       return = returning_clause(prefix, table, returning, "UPDATE")
 
-      IO.iodata_to_binary(["UPDATE ", quote_table(prefix, table), " SET ",
-                           fields, " WHERE ", filters | return])
+      ["UPDATE ", quote_table(prefix, table), " SET ",
+       fields, " WHERE ", filters | return]
     end
 
     def delete(prefix, table, filters, returning) do
       {filters, _} = intersperse_reduce(filters, " AND ", 1, fn field, acc ->
         {[quote_name(field), " = ?" | Integer.to_string(acc)], acc + 1}
       end)
 
-      IO.iodata_to_binary(["DELETE FROM ", quote_table(prefix, table), " WHERE ",
-                           filters | returning_clause(prefix, table, returning, "DELETE")])
+      ["DELETE FROM ", quote_table(prefix, table), " WHERE ",
+       filters | returning_clause(prefix, table, returning, "DELETE")]
     end
 
     ## Query generation
@@ -328,13 +327,10 @@ if Code.ensure_loaded?(Sqlitex.Server) do
       quote_qualified_name(field, sources, idx)
     end
 
-    defp expr({:&, _, [idx, fields, _counter]}, sources, query) do
-      {source, name, schema} = elem(sources, idx)
-      if is_nil(schema) and is_nil(fields) do
-        error!(query, "SQLite does not support selecting all fields from #{source} without a schema. " <>
+    defp expr({:&, _, [idx]}, sources, query) do
+      {source, _name, _schema} = elem(sources, idx)
+      error!(query, "SQLite does not support selecting all fields from #{source} without a schema. " <>
                       "Please specify a schema or specify exactly which fields you want to select")
-      end
-      intersperse_map(fields, ", ", &[name, ?. | quote_name(&1)])
     end
 
     defp expr({:in, _, [left, right]}, sources, query) when is_list(right) do
@@ -363,8 +359,8 @@ if Code.ensure_loaded?(Sqlitex.Server) do
       ["NOT (", expr(expr, sources, query), ?)]
     end
 
-    defp expr(%Ecto.SubQuery{query: query, fields: fields}, _sources, _query) do
-      query.select.fields |> put_in(fields) |> all()
+    defp expr(%Ecto.SubQuery{query: query}, _sources, _query) do
+      all(query)
     end
 
     defp expr({:fragment, _, [kw]}, _sources, query) when is_list(kw) or tuple_size(kw) == 3 do
@@ -476,10 +472,11 @@ if Code.ensure_loaded?(Sqlitex.Server) do
     # transaction and trigger. See corresponding code in Sqlitex.
 
     defp returning(%Query{select: nil}, _sources, _cmd), do: []
-    defp returning(%Query{select: %{fields: [{:&, [], [_, fields, _]}]}}, sources, cmd) do
+    defp returning(%Query{select: %{fields: field_tuples}}, sources, cmd) do
       cmd = cmd |> Atom.to_string |> String.upcase
       table = table_from_first_source(sources)
-      fields = Enum.map_join([table | fields], ",", &quote_id/1)
+      fields = Enum.map_join([table | Enum.map(field_tuples, &field_from_field_tuple/1)],
+                             ",", &quote_id/1)
       [@pseudo_returning_statement, cmd, ?\s, fields]
     end
 
@@ -491,6 +488,8 @@ if Code.ensure_loaded?(Sqlitex.Server) do
       |> String.trim("\"")
     end
 
+    defp field_from_field_tuple({{:., [], [{:&, [], [0]}, f]}, [], []}), do: f
+
     defp returning_clause(_prefix, _table, [], _cmd), do: []
     defp returning_clause(prefix, table, returning, cmd) do
       fields = Enum.map_join([{prefix, table} | returning], ",", &quote_id/1)
@@ -741,11 +740,15 @@ if Code.ensure_loaded?(Sqlitex.Server) do
       do: [" DEFAULT '", escape_string(literal), ?']
     defp default_expr({:ok, literal}, _type) when is_number(literal) or is_boolean(literal),
       do: [" DEFAULT ", to_string(literal)]
+    defp default_expr({:ok, %{} = map}, :map) do
+      default = Ecto.Adapter.json_library().encode!(map)
+      [" DEFAULT ", single_quote(default)]
+    end
     defp default_expr({:ok, {:fragment, expr}}, _type),
       do: [" DEFAULT ", expr]
     defp default_expr({:ok, expr}, type),
       do: raise(ArgumentError, "unknown default `#{inspect expr}` for type `#{inspect type}`. " <>
-                               ":default may be a string, number, boolean, empty list or a fragment(...)")
+                               ":default may be a string, number, boolean, empty list, map (when type is Map), or a fragment(...)")
     defp default_expr(:error, _),
       do: []
 
@@ -765,6 +768,7 @@ if Code.ensure_loaded?(Sqlitex.Server) do
     # precision regardless of the declared column type. Decimals are the
     # only exception.
     defp column_type(:serial, _opts), do: "INTEGER"
+    defp column_type(:bigserial, _opts), do: "INTEGER"
     defp column_type(:string, _opts), do: "TEXT"
     defp column_type(:map, _opts), do: "TEXT"
     defp column_type({:map, _}, _opts), do: "TEXT"
@@ -794,6 +798,7 @@ if Code.ensure_loaded?(Sqlitex.Server) do
       do: quote_name(name)
 
     defp reference_column_type(:serial, _opts), do: "INTEGER"
+    defp reference_column_type(:bigserial, _opts), do: "INTEGER"
     defp reference_column_type(type, opts), do: column_type(type, opts)
 
     defp reference_on_delete(:nilify_all), do: " ON DELETE SET NULL"
@@ -843,6 +848,8 @@ if Code.ensure_loaded?(Sqlitex.Server) do
       [?", name, ?"]
     end
 
+    defp single_quote(value), do: [?', escape_string(value), ?']
+
     defp intersperse_map(list, separator, mapper, acc \\ [])
     defp intersperse_map([], _separator, _mapper, acc),
       do: acc
@@ -877,8 +884,5 @@ if Code.ensure_loaded?(Sqlitex.Server) do
     defp error!(query, message) do
       raise Ecto.QueryError, query: query, message: message
     end
-
-    # Use Ecto's JSON library (currently Poison) for embedded JSON datatypes.
-    defp json_library, do: Application.get_env(:ecto, :json_library)
   end
 end