When we only have an output and no input of our own, filter.run() seems
weird to call, especially since it'll only be closing a handle and waiting
for fast-import to finish. Add a finish() synonym for such a case to make
external code callers more legible.
Signed-off-by: Elijah Newren <newren@gmail.com>
This will allow exporting from one repo into a different repo, and
combined with chained RepoFilter instances from commit 81016821a1
(filter-repo: allow chaining of RepoFilter instances, 2019-01-07), will
even allow things like splicing separate repositories together.
Signed-off-by: Elijah Newren <newren@gmail.com>
We do not want to kill fast-import processes unused; it's better
to abort before those processes are created when we know we need to.
Signed-off-by: Elijah Newren <newren@gmail.com>
Allow each instance to be just input or just output so that we can splice
repos together or split one into multiple different repos.
Signed-off-by: Elijah Newren <newren@gmail.com>
If we are using --stdin, it should be okay to import into a bare repo,
but the checks were enforcing that we were in a clone with a packfile.
Relax the check to work within a bare repo as well.
Signed-off-by: Elijah Newren <newren@gmail.com>
If we have blob callbacks, we cannot pass --no-data to fast-export. Also,
with blob callbacks, any file the callback modifies could match the
modification done to the file by a subsequent commit, possibly making the
later commit empty. As such, we keep a record of all filenames modified
(by blob or commit callbacks), and then check all these filenames for all
subsequent commits to see if it causes empty commits. In particular, if
files other than these are modified in a non-merge commit, we know that
the commit will not become empty so we can bypass the empty-pruning
checks.
Signed-off-by: Elijah Newren <newren@gmail.com>
If a commit was a non-merge commit previously, then since we do not do
any kind of blob modifications (or funny parent grafting), there is no
way for a filemodify instruction to introduce the same version of the
file that already existed in the parent, as such the only check we need
to do to determine whether a commit becomes empty is whether
file_changes is empty. Subsequent more expensive checks can be skipped.
Signed-off-by: Elijah Newren <newren@gmail.com>
Split a lot of the logic out into separate functions, and avoid
flattening parents when the original commit history itself had
redundant parents (such as --no-ff merges).
Signed-off-by: Elijah Newren <newren@gmail.com>
commits may not have any parents at all. As such,
parse_optional_parent_ref() is used expecting that it will sometimes
return None.
Now, when commits are skipped, we have a scheme to translate anyone that
depends on such commits to instead depend on the nearest ancestor of
such commits. If the entire ancestry of a commit was skipped along with
a comit, then that commit will be translated to None, which is
indistinguishable from there having been no parent to begin with.
Sometimes our scheme needs to distinguish between a commit that started
with no parents and one which ended up with no parents, so we need a way
to tell these apart.
Also, not knowing the original parent makes it hard for us to
determine if the original had the same weird topology that the current
commit does. For example, it is possible for a merge commit to have
one parent be the ancestor of another (particularly when --no-ff is
passed to git merge), or even for a merge commit to have the same
commit used as both parents (if you use low-level commands to create
a crazy commit). There are cases where the pruning of some commits
could cause either of these situations to arise, and it's useful to be
able to distinguish between intentionally "weird" history and history
that has been made weird due to other pruning, because the latter we
may have reason to do additional pruning on.
Signed-off-by: Elijah Newren <newren@gmail.com>
Oh, boy, timezone +051800 exists in the wild. Is that 0518 hours and 00
minutes? Or 05 hours and 1800 minutes? Or 051 hours and 800 minutes?
Attempt to do something sane with these broken commits that fast-import
barfs on. Also, fix an old bug in the handling of ahead-of-UTC timezones.
Signed-off-by: Elijah Newren <newren@gmail.com>
Apparently, the default for subprocess stdout is unbuffered; switching
it to buffered yields a huge 40% speedup. Doing this also exposes the
need to add fi_input.flush() calls, highlighting another performance
issue. We may be able to have fewer such calls with some refactoring,
but that is a bigger separate change. Just having them highlighted to
remind about them as a performance issue is good for now.
Signed-off-by: Elijah Newren <newren@gmail.com>
As suggested by Peff, use rev-list & diff-tree to get the information we
need, instead of relying on fast-export (with some out-of-tree patches)
to get that information.
Signed-off-by: Elijah Newren <newren@gmail.com>
If a tag points at a commit whose changes are all filtered out and thus
becomes empty and gets pruned, and all of its ancestors are likewise
pruned, then there is no need for the tag; just nuke it.
Signed-off-by: Elijah Newren <newren@gmail.com>
This option walks through the repository history and creates a report
with basic statistics, rename related information, and sizes of objects
and when/if those have been deleted. It primarily looks at unpacked
sizes (i.e. size of object ignoring delta-ing and compression), and
sums the size of each version of the file for each path. Additionally,
it aggregates these sums by extension and by directory, and tracks
whether paths, extensions, and directories have been deleted. This can
be very useful in determining what the big things are, and whether they
might have been considered to have been mistakes to add to the
repository in the first place.
There are numerous caveats with the determination of "deleted" and
"renamed", and can give both false positives and false negatives. But
they are only meant as a helpful heuristic to give others a starting
point for an investigation, and the information provide so far is useful.
I do want to improve the equivalence classes (rename handling), but that
is for a future commit.
Signed-off-by: Elijah Newren <newren@gmail.com>
Commit messages often refer to past commits; while rewriting commits we
would also like to update these commit messages to refer to the new
commit names.
In the case that a commit message references another commit which was
dropped by the filtering process, we have no way to rewrite the commit
message to reference a valid commit hash. Instead of dying, note the
suboptimal commit in the suboptimal-issues file.
Signed-off-by: Elijah Newren <newren@gmail.com>
This will be used later to help with commit message rewriting (so that
commits can continue to refer to other commits in their history, using
the new rewritten hashes for those commits), and perhaps also in
removing blobs by id.
Signed-off-by: Elijah Newren <newren@gmail.com>
If ancient history that pre-dated some subdirectory had a few empty
commits, we would rather those all got pruned as well. Empty commits
from the original repository should only be retained if they have at
least one retained parent.
Signed-off-by: Elijah Newren <newren@gmail.com>
When the pruning of empty commits causes a culling of parents of a merge
commit, so that the merge commit drops to just one parent, the commit
likely becomes misleading since the commit is no longer a merge commit
but the message probably implies it is. (e.g. "Merge branch maint into
master"). There's nothing we can do to automatically fix this, but we
can note it as a suboptimal issue in the filtering process.
Signed-off-by: Elijah Newren <newren@gmail.com>
Our filtering process will rewrite (and drop) commits, causing refs to
also get updated. A useful debugging aid for users is to write metadata
showing the mapping from old commit IDs to new commit IDs, and from the
hash that old refs pointed and the hash that the new ones do.
Signed-off-by: Elijah Newren <newren@gmail.com>
In the previous commit, we detected when an entire line of history back to
a common ancestor of the merge became empty commits, and avoided having a
commit be merged with itself. This commits looks through the changes
specified in the commit, which are always specified relative to the first
parent, so that if the first parent side was the empty one we can still
detect if the merge commit adds no extra changes relative to its remaining
parent.
Signed-off-by: Elijah Newren <newren@gmail.com>
Pruning of empty commits can cause an entire line of history to become
empty and be pruned, resulting in a merge commit that merges some commit
with one of its ancestors. In such a case, we should remove the
unnecessary parent(s) -- which can and will often result in the merge
commit being empty so we can remove it as well.
Currently, if the side that becomes empty is the first parent side, then
we do not detect if the commit becomes empty, due to the way that
fast-export lists changes in a merge commit relative to first parent only.
A subsequent commit will address this.
Note that the callbacks could theoretically insert additional commits or
reparent our commit on top of something else, meaning that the ancestry
graph might need post-callback updates. However, in any extreme case
where that mattered, we would more or less need full updates to the
ancestry graph to be made for all the new commits from the callback as
well, and once we expect the callback to handle any ancestry graph
updates it can handle modifying it for the current commit. However, it
is hard to come up with a case where it matters, since for the most part
we just want to know whether our filtering causes commits to become
empty and knowing the source repo we are exporting from is sufficient
information without knowing any new commits inserted or reparenting that
happens elsewhere.
Signed-off-by: Elijah Newren <newren@gmail.com>
Use the 'feature done' ability to mark when the fast-import stream is
finished, so that an aborted run (due to running into some kind of bug
while filtering, whether a bug in the code, or an error in the repo or
flags specified for the case under consideration) won't cause the repo
to be rewritten.
Signed-off-by: Elijah Newren <newren@gmail.com>
It may be that the only time a reference is shown in the fast-export stream
is for a commit which will become empty due to the filtering. We do not
want such refs to be left out and thus not be updated; we want them to
instead be set to the nearest non-empty ancestor. Only if it has no
non-empty ancestor would we want it to be stripped out.
Signed-off-by: Elijah Newren <newren@gmail.com>
Slightly re-order the code to make input, output, and filtering sections
distinct. Also, avoid running `git fast-import` at all when we're in
--dry-run mode.
Signed-off-by: Elijah Newren <newren@gmail.com>
We always called record_id_rename with handle_transitivity set to True, and
I do not know of a use case that would do otherwise so let's just hardcode
that value.
Signed-off-by: Elijah Newren <newren@gmail.com>
My idea to use --export-marks and --import-marks to avoid the need for the
id_offset was not tested and apparently a bad idea. When splicing together
multiple repositories, the second will croak if we pass it --import-marks
with a file having sha1sums that don't exist in that repository.
I'm afraid this might conflict with the --import-marks stuff used in collab
so I've only enabled it for streams beyond the first. So there might be an
issue using --import-marks on a second or later fast-export output stream,
but I can't think of a use case for that...
Signed-off-by: Elijah Newren <newren@gmail.com>
Filtering input from multiple repositories can still be done; however, to
avoid overloading of mark numbers, one should pass --export-marks=<file>
to the first git fast-export and pass --import-marks=<file> to the second.
Signed-off-by: Elijah Newren <newren@gmail.com>
git-fast-import requires that file changes listed in a merge commit be
relative to the first parent. Thus, if I've added new files on a branch
being merged in from the second or later parents, I need to manually
modify the list of files in the merge commit as well. In order to do that,
as soon as I splice in any commit, I have to record the list of new files
for both that commit and every descendant it has.
Signed-off-by: Elijah Newren <newren@gmail.com>
Also, provide an OutputStream class, to make it easy to still direct all
output to some file rather than always sending to git fast-import.
Signed-off-by: Elijah Newren <newren@gmail.com>
Automatically do renaming of references to commits that were skipped, and
automatically remove skipped blobs from the output of commits that
reference them.
Signed-off-by: Elijah Newren <newren@gmail.com>
Explicitly specify --topo-order; git-fast-export fails on some topologies
unless it traverses in topological order.
Signed-off-by: Elijah Newren <newren@gmail.com>
pyparsing sucks a whole file into memory at a time and then parses, which
is really bad in this case since the output from git-fast-export is huge.
I entered disk swapping madness pretty easily. So, now I just do my own
manual parsing.
Signed-off-by: Elijah Newren <newren@gmail.com>
* Allow hooking up (and filtering) multiple git fast-export's to one import
* Allow user callbacks to force dumping of object in order to reference it
with subsequent inserted objects
* Put the separate callbacks and global vars in the calling program into a
combined class
Signed-off-by: Elijah Newren <newren@gmail.com>
My prior handlings of marks would only work if there were not additions
or removals from the fast-export stream. Further, I referred to these as
marks even though I really only accept idnum values, not sha1s or anything
else. So, now I refer to these as ids everywhere, and I am much more
careful in my handling of ids.
Signed-off-by: Elijah Newren <newren@gmail.com>
The commit_callback call was trying to pass a Reset object, which was
not defined. Copy-n-paste-n-forget-to-replace isn't good. Now it passes
a Commit object.
Signed-off-by: Elijah Newren <newren@gmail.com>
We still only parse a single blob, but this should put the infrastructure
in place for parsing more output from git-fast-export.
Signed-off-by: Elijah Newren <newren@gmail.com>
Elijah Newren
James Foucar