Commit Graph

10 Commits (1aef90ce19c7d1c101e4e80a637f0052f617c12b)

Author SHA1 Message Date
Romain 1aef90ce19 fix: make stats displayed during indexing more explicit
Before this, the "total" value represented files on disk
which was confusing in partial and incremental modes
5 months ago
Romain 3590b15cfb chore: use node prefix for all builtin modules 6 months ago
Romain 8a43cebc72 feat: support multiple scan modes for partial gallery generation 1 year ago
Romain 8cb7a3596b chore: upgrade to latest version of StandardJS 1 year ago
Romain 9425604dd7 chore(dependencies): upgrade Standard from v11 to v12 which introduces a style change 6 years ago
Romain 7dd714c957 refactor(tests): dynamically generate test folders to check more edge cases 6 years ago
Romain 91528ad076 test(index): cleanup thumbsup.db between test runs, otherwise the test does nothing 7 years ago
Romain 3e098e2328 Extract Handlebars helpers to prepare for theme structure + add unit tests 7 years ago
Romain cbf8a1f3b4 Don’t index media from Synology thumbnail folders (@eaDir) nor folders starting with “.” 7 years ago
Romain 24b2f9bd7c Major optimisations: SQLite index + faster disk glob + new exiftool streaming
1. Move from a JSON index to a SQLite database.
  - This allows the indexing to be interrupted & resumed
  - Updating the index consumes less RAM than loading / saving an entire JSON object
  - Loading the index consumes less RAM since it can be streamed, only exacting the properties we need every time (instead of loading all EXIF data in memory, only to discard most of it later)
  - These make a big difference when processing 10,000+ photos

2. Switch from <glob> to a manual <readdir>
  - Glob would take several hundred or GB of RAM when asked to find several thousand files
  - Manual approach with <micromatch> library does the same thing in a fraction of the time / memory usage

3. Exiftool optimisations
  - Run 1 exiftool process per CPU, still in batch mode (divide all files to be read into 1 bucket per CPU)
  - Stream the exiftool output instead of buffering it in memory
7 years ago