Absolutely incorrect, fair use applies to *reproducing* small volumes of work, not analyzing it. If I published an article gleaning some conclusion based on an analysis of 10,000 issues of the New York Times, that would still 100% be fair use; similarly, Google is absolutely allowed to publish word count metrics based on their scanned book repo, even though publishing the books themselves is not fair use. You are trying to read something into the fair use doctrine that is absolutely not there (to the extent that anything is there, which very little is other than "I'll know it when I see it" and prior case law, unfortunately).

When I fair use a small quote from a book, I may have read the whole book.

Now I'll go the other way and wonder if it should still fall under fair use if I respond to requests for small quotes programmatically and eventually quote the entire book.

Or here is the real analagous question:

Fair use is about more than just the size of the excerpt.

If you write an article about good writing, and quote a choice paragraph from someone else's work to show an example, and credit that quote, that is fair use.

Is it fair use if you read an awesome paragraph, something that really is the result of the authors unique intellect and effort and craftsmanship, and makes you think "damn", and then drop that same jewel into your book?

The difference is, the paragraph isn't being included for examination or comment or transformation, it's being included to directly copy and perform it's original function as part of what makes a work a great work, and, it's not being credited in any bibliography or footnotes or directly.

The reader reads the paragraph and is impressed by your deep insight, which you never had, and the original author did.

I think all in all, this sort of copying & re-use should be allowed to happen somehow, because software is more like a machine than a novel, and humanity benefits when machines work well. There just needs to be some sort of rules around it about what gets included in the training sets and how both the input and the output are credited and acknowleged.

Right now, I think Github are simply outlaws. 100% of the output is violating the copyright of the code in the training set, because 100% of the input is copyrighted one way or another and none of it is being declared on the output. And it's allowing incompatiple sources to mix and the origonal terms to be stripped. The training set includes both proprietary and open source, and the output is being used in both proprietary and open source.

And there is no way that Github does not have this same understanding that I just described. I refuse to believe I am that special that I can see this and no one at Github did.

So they are not merely possibly inadvertant outlaws, they are deliberate knowing intentional outlaws.

I think a key thing here is your identification of a paragraph. Nobody would think to exert copyright over individual words. Phrases and epigrams are considered worthy of attribution, but only in exceptional cases. Copying sentences is starting to get into plagiarism, though single sentences would usually be forgiven because noting or remembering a single sentence while forgetting the source is an easy mistake to make. Copying whole paragraph, by contrast, is unlikely to be casual.

I think in programming therms a useful parallel might be copying at the module rather than the statement or function level. For example, if I write some code prompts to do the following:

  - validate my API key with Twitter
  - solicit the input of a Twitter username
  - download the up to 500 of that user's tweets
  - convert the json to a dataframe
  - plot the derivative of the intervals between tweets

...many of those tasks can be fairly described as helper functions, either taken directly from documentation (like interfacing with an API) or being so elementary as to be generic. If any one of these tasks happened to come from your code or mine, and the rest from other programs, it wouldn't feel like much of an infringement. If all of them came from the same body of code, it would.