When we only have an output and no input of our own, filter.run() seems
weird to call, especially since it'll only be closing a handle and waiting
for fast-import to finish. Add a finish() synonym for such a case to make
external code callers more legible.
Signed-off-by: Elijah Newren <newren@gmail.com>
This will allow exporting from one repo into a different repo, and
combined with chained RepoFilter instances from commit 81016821a1
(filter-repo: allow chaining of RepoFilter instances, 2019-01-07), will
even allow things like splicing separate repositories together.
Signed-off-by: Elijah Newren <newren@gmail.com>
We do not want to kill fast-import processes unused; it's better
to abort before those processes are created when we know we need to.
Signed-off-by: Elijah Newren <newren@gmail.com>
Allow each instance to be just input or just output so that we can splice
repos together or split one into multiple different repos.
Signed-off-by: Elijah Newren <newren@gmail.com>
If we are using --stdin, it should be okay to import into a bare repo,
but the checks were enforcing that we were in a clone with a packfile.
Relax the check to work within a bare repo as well.
Signed-off-by: Elijah Newren <newren@gmail.com>
If we have blob callbacks, we cannot pass --no-data to fast-export. Also,
with blob callbacks, any file the callback modifies could match the
modification done to the file by a subsequent commit, possibly making the
later commit empty. As such, we keep a record of all filenames modified
(by blob or commit callbacks), and then check all these filenames for all
subsequent commits to see if it causes empty commits. In particular, if
files other than these are modified in a non-merge commit, we know that
the commit will not become empty so we can bypass the empty-pruning
checks.
Signed-off-by: Elijah Newren <newren@gmail.com>
If a commit was a non-merge commit previously, then since we do not do
any kind of blob modifications (or funny parent grafting), there is no
way for a filemodify instruction to introduce the same version of the
file that already existed in the parent, as such the only check we need
to do to determine whether a commit becomes empty is whether
file_changes is empty. Subsequent more expensive checks can be skipped.
Signed-off-by: Elijah Newren <newren@gmail.com>
Split a lot of the logic out into separate functions, and avoid
flattening parents when the original commit history itself had
redundant parents (such as --no-ff merges).
Signed-off-by: Elijah Newren <newren@gmail.com>
commits may not have any parents at all. As such,
parse_optional_parent_ref() is used expecting that it will sometimes
return None.
Now, when commits are skipped, we have a scheme to translate anyone that
depends on such commits to instead depend on the nearest ancestor of
such commits. If the entire ancestry of a commit was skipped along with
a comit, then that commit will be translated to None, which is
indistinguishable from there having been no parent to begin with.
Sometimes our scheme needs to distinguish between a commit that started
with no parents and one which ended up with no parents, so we need a way
to tell these apart.
Also, not knowing the original parent makes it hard for us to
determine if the original had the same weird topology that the current
commit does. For example, it is possible for a merge commit to have
one parent be the ancestor of another (particularly when --no-ff is
passed to git merge), or even for a merge commit to have the same
commit used as both parents (if you use low-level commands to create
a crazy commit). There are cases where the pruning of some commits
could cause either of these situations to arise, and it's useful to be
able to distinguish between intentionally "weird" history and history
that has been made weird due to other pruning, because the latter we
may have reason to do additional pruning on.
Signed-off-by: Elijah Newren <newren@gmail.com>
Oh, boy, timezone +051800 exists in the wild. Is that 0518 hours and 00
minutes? Or 05 hours and 1800 minutes? Or 051 hours and 800 minutes?
Attempt to do something sane with these broken commits that fast-import
barfs on. Also, fix an old bug in the handling of ahead-of-UTC timezones.
Signed-off-by: Elijah Newren <newren@gmail.com>
Apparently, the default for subprocess stdout is unbuffered; switching
it to buffered yields a huge 40% speedup. Doing this also exposes the
need to add fi_input.flush() calls, highlighting another performance
issue. We may be able to have fewer such calls with some refactoring,
but that is a bigger separate change. Just having them highlighted to
remind about them as a performance issue is good for now.
Signed-off-by: Elijah Newren <newren@gmail.com>
As suggested by Peff, use rev-list & diff-tree to get the information we
need, instead of relying on fast-export (with some out-of-tree patches)
to get that information.
Signed-off-by: Elijah Newren <newren@gmail.com>
If a tag points at a commit whose changes are all filtered out and thus
becomes empty and gets pruned, and all of its ancestors are likewise
pruned, then there is no need for the tag; just nuke it.
Signed-off-by: Elijah Newren <newren@gmail.com>
This option walks through the repository history and creates a report
with basic statistics, rename related information, and sizes of objects
and when/if those have been deleted. It primarily looks at unpacked
sizes (i.e. size of object ignoring delta-ing and compression), and
sums the size of each version of the file for each path. Additionally,
it aggregates these sums by extension and by directory, and tracks
whether paths, extensions, and directories have been deleted. This can
be very useful in determining what the big things are, and whether they
might have been considered to have been mistakes to add to the
repository in the first place.
There are numerous caveats with the determination of "deleted" and
"renamed", and can give both false positives and false negatives. But
they are only meant as a helpful heuristic to give others a starting
point for an investigation, and the information provide so far is useful.
I do want to improve the equivalence classes (rename handling), but that
is for a future commit.
Signed-off-by: Elijah Newren <newren@gmail.com>
Commit messages often refer to past commits; while rewriting commits we
would also like to update these commit messages to refer to the new
commit names.
In the case that a commit message references another commit which was
dropped by the filtering process, we have no way to rewrite the commit
message to reference a valid commit hash. Instead of dying, note the
suboptimal commit in the suboptimal-issues file.
Signed-off-by: Elijah Newren <newren@gmail.com>
This will be used later to help with commit message rewriting (so that
commits can continue to refer to other commits in their history, using
the new rewritten hashes for those commits), and perhaps also in
removing blobs by id.
Signed-off-by: Elijah Newren <newren@gmail.com>
If ancient history that pre-dated some subdirectory had a few empty
commits, we would rather those all got pruned as well. Empty commits
from the original repository should only be retained if they have at
least one retained parent.
Signed-off-by: Elijah Newren <newren@gmail.com>
When the pruning of empty commits causes a culling of parents of a merge
commit, so that the merge commit drops to just one parent, the commit
likely becomes misleading since the commit is no longer a merge commit
but the message probably implies it is. (e.g. "Merge branch maint into
master"). There's nothing we can do to automatically fix this, but we
can note it as a suboptimal issue in the filtering process.
Signed-off-by: Elijah Newren <newren@gmail.com>
Our filtering process will rewrite (and drop) commits, causing refs to
also get updated. A useful debugging aid for users is to write metadata
showing the mapping from old commit IDs to new commit IDs, and from the
hash that old refs pointed and the hash that the new ones do.
Signed-off-by: Elijah Newren <newren@gmail.com>
In the previous commit, we detected when an entire line of history back to
a common ancestor of the merge became empty commits, and avoided having a
commit be merged with itself. This commits looks through the changes
specified in the commit, which are always specified relative to the first
parent, so that if the first parent side was the empty one we can still
detect if the merge commit adds no extra changes relative to its remaining
parent.
Signed-off-by: Elijah Newren <newren@gmail.com>
Pruning of empty commits can cause an entire line of history to become
empty and be pruned, resulting in a merge commit that merges some commit
with one of its ancestors. In such a case, we should remove the
unnecessary parent(s) -- which can and will often result in the merge
commit being empty so we can remove it as well.
Currently, if the side that becomes empty is the first parent side, then
we do not detect if the commit becomes empty, due to the way that
fast-export lists changes in a merge commit relative to first parent only.
A subsequent commit will address this.
Note that the callbacks could theoretically insert additional commits or
reparent our commit on top of something else, meaning that the ancestry
graph might need post-callback updates. However, in any extreme case
where that mattered, we would more or less need full updates to the
ancestry graph to be made for all the new commits from the callback as
well, and once we expect the callback to handle any ancestry graph
updates it can handle modifying it for the current commit. However, it
is hard to come up with a case where it matters, since for the most part
we just want to know whether our filtering causes commits to become
empty and knowing the source repo we are exporting from is sufficient
information without knowing any new commits inserted or reparenting that
happens elsewhere.
Signed-off-by: Elijah Newren <newren@gmail.com>
Elijah Newren