fix: keep the line terminator of a file a save overwrites

.golangci.yml

@@ -50,6 +50,10 @@ linters:
     - wastedassign
     - whitespace
   settings:
+    goconst:
+      # A literal repeated across test cases is not a constant the package is
+      # missing, and counting them pushed production strings over the threshold.
+      ignore-tests: true
     errcheck:
       disable-default-exclusions: false
       check-type-assertions: true

CHANGELOG.md

@@ -5,6 +5,20 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Added
+
+- `DumpOptions.WithLineEnding` sets the line terminator of csv, tsv, and ltsv output, and a save that overwrites a file it loaded from a path takes that file's own terminator without being asked ([#269](https://github.com/nao1215/filesql/issues/269)). A save kept a source's compression and its text encoding but wrote every record with `\n`, so a CRLF file saved in place came back LF throughout: a caller who edited one row got a file whose every line had changed, which is a whole-file diff in a repository configured for CRLF and a file the tools reading it no longer saw as they had. The terminator is read from the file about to be replaced — through its codec, so a `.csv.gz` is read as the text inside it — and a file with mixed terminators keeps whichever the majority of its lines use, so one stray ending cannot rewrite the rest. The whole file is counted, through a fixed buffer rather than in memory, and a line break inside a quoted CSV field is field data rather than a terminator: a workbook-style export with CRLF between records and LF inside a quoted address is a CRLF file. A dump to a new destination writes `\n` unless `WithLineEnding(LineEndingCRLF)` says otherwise, which is what every save wrote before this existed. `parser.WriteTSVRecordLineEnding` is the same choice for a caller writing TSV records directly. Parquet and XLSX are not line-based and are unaffected.
+
+### Fixed
+
+- A `SIMILAR TO` pattern that ends in a backslash no longer matches the wrong thing. The translation wrote the trailing backslash through, where it escaped the anchor the translation appends: `a\` became the regular expression `^a\$`, which matches the literal text `a$` and not the backslash the pattern ends with. A trailing escape now escapes itself.
+
+### Changed
+
+- Dependencies: `github.com/moov-io/ach` 1.61.3 → 1.62.1, `github.com/moov-io/wire` 0.15.8 → 0.15.9, `modernc.org/sqlite` 1.55.0 → 1.56.0, `github.com/klauspost/compress` 1.19.1 → 1.19.2, `github.com/pierrec/lz4/v4` 4.1.27 → 4.1.28.
+
 ## [0.43.1] - 2026-08-09
 
 ### Fixed

README.md

@@ -403,18 +403,21 @@ Changes live in memory until you save them.
 - `EnableAutoSave` saves when `db.Close()` runs.
 - `EnableAutoSaveOnCommit` saves after each committed transaction.
 
-`DumpOptions` decides the format, the compression, and the text encoding of csv, tsv, and ltsv output:
+`DumpOptions` decides the format, the compression, the text encoding, and the line terminator of csv, tsv, and ltsv output:
 
 ```go
 options := filesql.NewDumpOptions().
 	WithFormat(filesql.OutputFormatCSV).
-	WithEncoding(filesql.EncodingShiftJIS)
+	WithEncoding(filesql.EncodingShiftJIS).
+	WithLineEnding(filesql.LineEndingCRLF)
 
 err := filesql.DumpDatabase(db, "./output", options)
 ```
 
 Output is UTF-8 unless `WithEncoding` says otherwise, which is what a save wrote before the option existed. `EncodingShiftJIS`, `EncodingEUCJP`, `EncodingISO2022JP`, `EncodingUTF16LE`, and `EncodingUTF16BE` are the others; the UTF-16 pair write a byte-order mark, so the read side recognizes them without being told. A value the encoding has no way to write fails the save with `ErrEncoding` and leaves the destination as it was, rather than being replaced with a substitute character — the same answer the read side gives to bytes it cannot decode. Parquet and XLSX carry their own encoding and ignore the option.
 
+Records end with `\n` unless `WithLineEnding` says otherwise. A save that overwrites a file it loaded from a path does not need to be told: it reads the terminator the file already uses and writes the same one, so a CRLF file edited in place stays CRLF and the rows nobody touched stay byte-identical. A file with mixed terminators keeps whichever one the majority of its lines use. Parquet and XLSX are not line-based and ignore the option.
+
 ### Excel sheet visibility
 
 A workbook can hide a sheet, and a hidden sheet often holds the spreadsheet's own working-out rather than data anyone meant to publish. filesql loads every sheet by default, hidden or not, so existing programs keep the tables they have.
@@ -472,7 +475,7 @@ The GoDoc examples are fully tested with `go test`. The tables below show the fa
 | Attach your own logger | `ExampleDBBuilder_WithLogger`, `ExampleNewSlogAdapter` | [example_api_test.go](./example_api_test.go) |
 | Open a read-only wrapper | `ExampleDBBuilder_OpenReadOnly` | [example_api_test.go](./example_api_test.go) |
 | Save on close or commit | `ExampleDBBuilder_EnableAutoSave`, `ExampleDBBuilder_EnableAutoSaveOnCommit`, `ExampleDBBuilder_DisableAutoSave` | [example_api_test.go](./example_api_test.go), [example_test.go](./example_test.go) |
-| Export tables with format/compression/encoding options | `ExampleDumpDatabase`, `ExampleNewDumpOptions`, `ExampleDumpOptions_WithFormat`, `ExampleDumpOptions_WithCompression`, `ExampleDumpOptions_WithEncoding` | [example_api_test.go](./example_api_test.go), [example_test.go](./example_test.go) |
+| Export tables with format/compression/encoding/line-ending options | `ExampleDumpDatabase`, `ExampleNewDumpOptions`, `ExampleDumpOptions_WithFormat`, `ExampleDumpOptions_WithCompression`, `ExampleDumpOptions_WithEncoding`, `ExampleDumpOptions_WithLineEnding` | [example_api_test.go](./example_api_test.go), [example_test.go](./example_test.go) |
 | Work with compression helpers directly | `ExampleNewCompressionHandler`, `ExampleNewCompressionFactory`, `ExampleCompressionFactory_DetectCompressionType` | [example_api_test.go](./example_api_test.go) |
 | Strip compression suffixes and inspect file types | `ExampleCompressionFactory_RemoveCompressionExtension`, `ExampleCompressionFactory_GetBaseFileType` | [example_api_test.go](./example_api_test.go) |
 | Inspect the malformed-row policy | `ExampleMalformedRowPolicy_String` | [example_api_test.go](./example_api_test.go) |

autosave_line_ending_test.go

@@ -0,0 +1,131 @@
+package filesql
+
+import (
+	"compress/gzip"
+	"io"
+	"os"
+	"path/filepath"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// TestAutoSaveOverwriteKeepsLineEnding pins that a save in place writes back the
+// terminator the file already used.
+//
+// It did not: every record was written with "\n" whatever the source used, so a
+// CRLF file came back LF throughout. A caller who edited one row got a file
+// whose every line had changed — a whole-file diff in a repository configured
+// for CRLF, and a file the tools that read it no longer saw as they had.
+func TestAutoSaveOverwriteKeepsLineEnding(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name    string
+		file    string
+		content string
+		update  string
+		want    string
+	}{
+		{
+			name:    "CSV keeps CRLF",
+			file:    "crlf.csv",
+			content: "id,v\r\n1,a\r\n2,b\r\n",
+			update:  "UPDATE crlf SET v='x' WHERE id=1",
+			want:    "id,v\r\n1,x\r\n2,b\r\n",
+		},
+		{
+			name:    "CSV keeps LF",
+			file:    "lf.csv",
+			content: "id,v\n1,a\n2,b\n",
+			update:  "UPDATE lf SET v='x' WHERE id=1",
+			want:    "id,v\n1,x\n2,b\n",
+		},
+		{
+			name:    "TSV keeps CRLF",
+			file:    "crlf.tsv",
+			content: "id\tv\r\n1\ta\r\n2\tb\r\n",
+			update:  "UPDATE crlf SET v='x' WHERE id=1",
+			want:    "id\tv\r\n1\tx\r\n2\tb\r\n",
+		},
+		{
+			name:    "LTSV keeps CRLF",
+			file:    "crlf.ltsv",
+			content: "id:1\tv:a\r\nid:2\tv:b\r\n",
+			update:  "UPDATE crlf SET v='x' WHERE id=1",
+			want:    "id:1\tv:x\r\nid:2\tv:b\r\n",
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			path := filepath.Join(t.TempDir(), tt.file)
+			require.NoError(t, os.WriteFile(path, []byte(tt.content), 0o600))
+
+			require.NoError(t, autoSaveOverwrite(t, []string{path}, tt.update))
+
+			got, err := os.ReadFile(path) //nolint:gosec // Test path from t.TempDir()
+			require.NoError(t, err)
+			assert.Equal(t, tt.want, string(got), "only the edited row may differ from what was there")
+		})
+	}
+}
+
+// TestAutoSaveOverwriteKeepsLineEndingUnderCompression checks that the
+// terminator is read from the bytes inside the codec, not from the archive.
+func TestAutoSaveOverwriteKeepsLineEndingUnderCompression(t *testing.T) {
+	t.Parallel()
+
+	path := filepath.Join(t.TempDir(), "crlf.csv.gz")
+	file, err := os.Create(path) //nolint:gosec // Test path from t.TempDir()
+	require.NoError(t, err)
+	gz := gzip.NewWriter(file)
+	_, err = gz.Write([]byte("id,v\r\n1,a\r\n2,b\r\n"))
+	require.NoError(t, err)
+	require.NoError(t, gz.Close())
+	require.NoError(t, file.Close())
+
+	require.NoError(t, autoSaveOverwrite(t, []string{path}, "UPDATE crlf SET v='x' WHERE id=1"))
+
+	reopened, err := os.Open(path) //nolint:gosec // Test path from t.TempDir()
+	require.NoError(t, err)
+	defer reopened.Close()
+	reader, err := gzip.NewReader(reopened)
+	require.NoError(t, err)
+	defer reader.Close()
+
+	decompressed, err := io.ReadAll(reader)
+	require.NoError(t, err)
+	assert.Equal(t, "id,v\r\n1,x\r\n2,b\r\n", string(decompressed))
+}
+
+// TestDumpDatabase_WithLineEnding covers the option on a dump to a new
+// destination, where there is no existing file to take the terminator from.
+func TestDumpDatabase_WithLineEnding(t *testing.T) {
+	t.Parallel()
+
+	source := filepath.Join(t.TempDir(), "users.csv")
+	require.NoError(t, os.WriteFile(source, []byte("id,v\n1,a\n"), 0o600))
+
+	db, err := Open(source)
+	require.NoError(t, err)
+	t.Cleanup(func() { _ = db.Close() })
+
+	outputDir := t.TempDir()
+	require.NoError(t, DumpDatabase(db, outputDir, NewDumpOptions().WithLineEnding(LineEndingCRLF)))
+
+	got, err := os.ReadFile(filepath.Join(outputDir, "users.csv")) //nolint:gosec // Test path from t.TempDir()
+	require.NoError(t, err)
+	assert.Equal(t, "id,v\r\n1,a\r\n", string(got))
+}
+
+// TestNewDumpOptions_DefaultsToLF pins the default, which is what a save wrote
+// before the option existed.
+func TestNewDumpOptions_DefaultsToLF(t *testing.T) {
+	t.Parallel()
+
+	assert.Equal(t, LineEndingLF, NewDumpOptions().LineEnding)
+	assert.Equal(t, LineEndingCRLF, NewDumpOptions().WithLineEnding(LineEndingCRLF).LineEnding)
+}

dialect/operators.go

@@ -392,11 +392,17 @@ func similarToRegexp(pattern string) string {
 			b.WriteString("\\")
 			b.WriteByte(c)
 		case '\\':
-			b.WriteByte(c)
 			if i+1 < len(pattern) {
+				b.WriteByte(c)
 				i++
 				b.WriteByte(pattern[i])
+				break
 			}
+			// A pattern that ends in an escape has nothing to escape. Written
+			// through, the backslash would escape the anchor this appends instead,
+			// so "a\" became ^a\$ — a regex matching a literal "$" rather than the
+			// backslash the pattern ends with.
+			b.WriteString(`\\`)
 		default:
 			b.WriteByte(c)
 		}

dialect/rewrite_edge_test.go

@@ -0,0 +1,132 @@
+package dialect
+
+import (
+	"errors"
+	"strings"
+	"testing"
+)
+
+// TestTranslate_LeavesUnrecognizedFormsAlone covers the rewrite rules' "not this
+// form" answers. A call that only looks like the one a rule handles is passed
+// through unchanged: rewriting it on a guess would turn a query the backend
+// understands into one it does not, and the caller never wrote the rewritten
+// form.
+func TestTranslate_LeavesUnrecognizedFormsAlone(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name    string
+		dialect Dialect
+		sql     string
+	}{
+		{name: "EXTRACT without a part", dialect: PostgreSQL, sql: `SELECT EXTRACT(x) FROM t`},
+		{name: "EXTRACT without FROM", dialect: PostgreSQL, sql: `SELECT EXTRACT(year x) FROM t`},
+		{name: "CAST without AS", dialect: PostgreSQL, sql: `SELECT CAST(x) FROM t`},
+		{name: "CAST to something that is not a type name", dialect: PostgreSQL, sql: `SELECT CAST(x AS 3) FROM t`},
+		{name: "CAST to a type this package does not know", dialect: PostgreSQL, sql: `SELECT CAST(x AS quux) FROM t`},
+		{name: "DATE_ADD without a second argument", dialect: MySQL, sql: `SELECT DATE_ADD(d) FROM t`},
+		{name: "DATE_ADD without INTERVAL", dialect: MySQL, sql: `SELECT DATE_ADD(d, 3) FROM t`},
+		{name: "SIMILAR without TO", dialect: PostgreSQL, sql: `SELECT x SIMILAR t FROM t`},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			got, err := Translate(tt.dialect, tt.sql)
+			if err != nil {
+				t.Fatalf("Translate(%q) error: %v", tt.sql, err)
+			}
+			if got != tt.sql {
+				t.Fatalf("Translate(%q) = %q, want it unchanged", tt.sql, got)
+			}
+		})
+	}
+}
+
+// TestTranslate_RefusesIntervalsItCannotRepresent covers the INTERVAL forms that
+// are recognized and cannot be carried out. Each is refused by name rather than
+// dropped, because an interval silently left out of a query answers with rows
+// from the wrong dates.
+func TestTranslate_RefusesIntervalsItCannotRepresent(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name string
+		sql  string
+		want string
+	}{
+		{
+			name: "no unit",
+			sql:  `SELECT DATE_ADD(d, INTERVAL 1) FROM t`,
+			want: "missing a unit",
+		},
+		{
+			name: "a unit no dialect defines",
+			sql:  `SELECT DATE_ADD(d, INTERVAL 1 FORTNIGHT) FROM t`,
+			want: "unsupported INTERVAL unit",
+		},
+		{
+			name: "no value",
+			sql:  `SELECT DATE_ADD(d, INTERVAL DAY) FROM t`,
+			want: "missing a value",
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			_, err := Translate(MySQL, tt.sql)
+			if !errors.Is(err, ErrUnsupportedSyntax) {
+				t.Fatalf("Translate(%q) error = %v, want ErrUnsupportedSyntax", tt.sql, err)
+			}
+			if !strings.Contains(err.Error(), tt.want) {
+				t.Fatalf("Translate(%q) error = %q, want it to mention %q", tt.sql, err, tt.want)
+			}
+		})
+	}
+}
+
+// TestTranslate_CastKeepsTypeParameters checks that a parameterized type reaches
+// the cast helper whole. Dropping the parameters would turn CHAR(3) into CHAR
+// and DECIMAL(10,2) into DECIMAL, so a value would be cast to a different type
+// than the one the query names.
+func TestTranslate_CastKeepsTypeParameters(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name string
+		sql  string
+		want string
+	}{
+		{name: "a length", sql: `SELECT CAST(x AS CHAR(3)) FROM t`, want: `'CHAR(3)'`},
+		{name: "a precision and scale", sql: `SELECT CAST(x AS DECIMAL(10,2)) FROM t`, want: `'DECIMAL(10,2)'`},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			got, err := Translate(PostgreSQL, tt.sql)
+			if err != nil {
+				t.Fatalf("Translate(%q) error: %v", tt.sql, err)
+			}
+			if !strings.Contains(got, tt.want) {
+				t.Fatalf("Translate(%q) = %q, want it to carry %s", tt.sql, got, tt.want)
+			}
+		})
+	}
+}
+
+// TestTranslate_DateSubNegatesTheAmount pins that subtracting an interval is the
+// same helper with the amount negated, so the month clamping is applied in both
+// directions rather than only when adding.
+func TestTranslate_DateSubNegatesTheAmount(t *testing.T) {
+	t.Parallel()
+
+	got, err := Translate(MySQL, `SELECT DATE_SUB(d, INTERVAL 1 MONTH) FROM t`)
+	if err != nil {
+		t.Fatalf("Translate error: %v", err)
+	}
+	if !strings.Contains(got, `interval_add(d, -(1), 'month')`) {
+		t.Fatalf("Translate = %q, want the amount negated through interval_add", got)
+	}
+}