Commit Graph

12 Commits (293cbf8469775dfd9162e8ae9314365273ab1bda)

Author SHA1 Message Date
Romain 293cbf8469 test: make all tests run on Linux + Windows 3 years ago
Romain 9bda0a4835 feat(database): new include/exclude arguments to filter the input folder 5 years ago
Romain ec6fbf9b0a test(database): more unit tests for file globbing patterns 5 years ago
Romain f30eddb6dd feat(options): new flags to selectively include photos / videos / raw photos
fixes #128
5 years ago
Felix Eckhofer 81a85004c2 Fix behavior for non-traversable directories (#108)
fix(index): Continue on errors during find

This ensures the gallery is created even when some files are
inaccessible (for example due to invalid names or file permissions).

Requires running the test in Docker as non-root to allow file access test
6 years ago
Romain be0ee25c64 test(index): don’t run the invalid filename test on macOS 6 years ago
Felix Eckhofer 124d1857b1 test(index): Add test for invalid filename behavior 6 years ago
Romain 81ee517062 feat(index): ignore files in a root #recycle folder (Synology recycle bin) 7 years ago
Romain 2a0a8d6631 Upgrade mock-fs to support Node 8 (includes dynamic require workaround) 7 years ago
Romain cbf8a1f3b4 Don’t index media from Synology thumbnail folders (@eaDir) nor folders starting with “.” 7 years ago
Romain ac520d40ad Fix case insensitive glob + add unit tests for globbing 7 years ago
Romain 24b2f9bd7c Major optimisations: SQLite index + faster disk glob + new exiftool streaming
1. Move from a JSON index to a SQLite database.
  - This allows the indexing to be interrupted & resumed
  - Updating the index consumes less RAM than loading / saving an entire JSON object
  - Loading the index consumes less RAM since it can be streamed, only exacting the properties we need every time (instead of loading all EXIF data in memory, only to discard most of it later)
  - These make a big difference when processing 10,000+ photos

2. Switch from <glob> to a manual <readdir>
  - Glob would take several hundred or GB of RAM when asked to find several thousand files
  - Manual approach with <micromatch> library does the same thing in a fraction of the time / memory usage

3. Exiftool optimisations
  - Run 1 exiftool process per CPU, still in batch mode (divide all files to be read into 1 bucket per CPU)
  - Stream the exiftool output instead of buffering it in memory
7 years ago