"GitHub Stars" is a very useful metric. But for *what*?

Question

Users on the GitHub website are able to "star" other people's repositories, thereby saving them in their list of Starred Repos. Some people use "stars" to indicate that they like a project, other people use them as bookmarks so they can follow what's going on with the repo later.

GitHub Stars are an easy metric to keep track of, and I've used them to measure how popular an open source project is. I admit to sometimes even using Stars to determine how battle-tested a repo is, under the assumption that a repo that has received more Stars must have received more use.

However, there are three limitations I noticed with GitHub stars (which makes my current approach to handling them pretty dubious):

• It is easy to create fake accounts on GitHub that can star repos. This can allow you to boost your metrics pretty easily, although you do not want to be obvious when doing them, lest the accounts get detected and deleted. (You can google for "buy GitHub stars" for those that are lazy technically). While you can use fake accounts to star your own repos, you may also want to star other people's repos to avoid leaving a paper trail. And since you don't want to be running all these fake accounts manually, you want to create bots and...you get the picture.
• GitHub Stars may be "juiced" by media attention, which is temporary and not actually based on sustained popularity. This is from personal experience -- after publicizing a fork of an my open source repo on Hacker News, I received some stars, but no corresponding increase in use of my fork. It's possible that a few people were interested in my repo, starred it, and then either never found an actual use case for my fork or promptly forgot about it later on.
• You can star a Repo for any reason. GitHub doesn't care. It merely notes that someone pressed the Star button and that's all. You can guess why it's starred (someone likes it or wants to look at it later), but other reasons are equally plausible (they are bored at work and like starring random repos, they want to star a repo of a friend of theirs).

I'm sure that metric means something. I'm not sure what though. What do GitHub Stars measure, and how should I use them to better understand the vitality/success/popularity of an open source project?

Any answer to this question should be either evidence-based or research-based.

Answer 1

Stars are exactly what you said they are. Some indication that someone (or maybe a bot) clicked on a button and taken in abstraction of anything else there is not much more that you can infer from that.

You could assume that everyone is using stars for the same purpose but as you pointed they could be used to show your appreciation of a project as well as for bookmarking it and therefore they are either some form of feedback to a repo as well as some form of bookkeeping for the star maker.

Star counts are still routinely used by researchers as a proxy for project popularity though while it may correlate OK for the most popular projects this likely does not hold for the vast number of other projects in the long tail.

So some may interpret it as a proxy for popularity but in in the end this likely a weak proxy. Combining the number of forks, watchers, issues, commits, etc. might be better in the end but still only a proxy and will never be an unbiased metric.

If you are in search of metric to determine if a project is popular, a better approach might be to determine who is reusing what. Which is another complex problem of its own especially since reuse may be private and hidden. And you could lace in other metrics but as always with a pinch of salt.

To illustrate this, take this ranking of users for my self. This correlate OK with my skills and interests in general but it does not know a little fact about me: I use forking as if these were bookmarks. So any metrics based on forks would be in my favor whether I care or not because of my personal and unique way to treat a fork. And in doing so, I also introduce a bias in any fork metrics for the projects I fork.

Answer 2

Research found out the "stargazers-based classifier [...] to exhibit high precision (97%) [in retrieving engineered software projects]". Source: Munaiah, N., Kroh, S., Cabrey, C. et al. Empir Software Eng (2017) 22: 3219. https://doi.org/10.1007/s10664-017-9512-6

We used reaper to measure the dimensions of 1,857,423 GitHub repositories. We then used manually classified data sets of repositories to train classifiers capable of predicting if a given GitHub repository contains an engineered software project. [...] The performance of the classifiers was evaluated using a set of 200 repositories with known ground truth classification. We also compared the performance of the classifiers to other approaches to classification (e.g. number of GitHub Stargazers) and found our classifiers to outperform existing approaches. We found stargazers-based classifier (with 10 as the threshold for number of stargazers) to exhibit high precision (97%) but an inversely proportional recall (32%). On the other hand, our best classifier exhibited a high precision (82%) and a high recall (86%). The stargazer-based criteria offers precision but fails to recall a significant portion of the population.

Definition 1 An engineered software project is a software project that leverages sound software engineering practices in one or more of its dimensions such as documentation, testing, and project management.

Answer 3

What do GitHub Stars measure

I take the stars as a measure of awareness, meaning that projects with lots of stars might be known to many people and projects with only a few stars may be relatively unknown.

I try to not read more into it but I infer weakly from the number of stars that projects with only a few stars might not be well tested, might not be very mature, might not be popular, while projects with a large number of stars might be well tested, might be mature and might be popular. In general, I will compare all projects above a certain flexible threshold of the number of stars and then decide completely independent of the number of stars which project is the best for me. In short, it's only a hint to me.

Answer 4

I think the general consensus here that stars are meaningless is a case of privileging theoretical a priori knowledge over empirical observation. True, the empirical case is sketchy and very low quality evidence, but I believe a lot of us can attest that our personal experience reflects a correlation between stars and significance of some kind; A significance that is usually popularity, which itself correlates with other good and bad traits. It is true that a lot of users might be using the stars in “non-standard” ways, but even that indicates some measure of popularity. And the important thing here is that the ecosystem is so large that it can tolerate a lot of noise. Even if %15 of users star completely randomly, it wouldn’t make the metric useless.

Of course, whenever there is a heuristical signal, there is effort to fake and cheat. This is seen, e.g., in animals that sport colors of poison, while being completely harmless. And still, the signals endure; Since for the poisonous animals to signal, merely a differential benefit of being eaten less is enough. For the predators, merely surviving more by not eating potentially poisonous animals is enough. And the cheaters are enough of a minority to be able to freeride the signal.

This is, on a calm meditation, obvious and observed repeatedly in nature, society and the cyberspace. Instagram’s likes are a great example. People like (and not like) for a dizzying breadth of reasons. The Instagram bot ecosystem is teeming. Nepotism is the default behavior. And yet, likes signify popularity (even if only to some niches). And the more likes, the bigger these niches. Sure, I myself hate most of the popular stuff on Instagram, but that is besides the point. The highly liked posts are not at all random; They push the human animal’s buttons methodically.

Of course, careful, good empirical investigations can shed more light on the correlations, but it’s dumb to reject something because you don’t understand its technical underpinnings. What’s the methodical research that shows you your brain’s conclusions are of any value?

We all need to accept that sometimes, we are under heavy technical debt. Ignoring obvious (though very imprecise and suspicious) empirical evidence is a frequent cause of intellectual folly.

Answer 5

I think Github star is like an OK and easy metric given there is not a lot of ways to tell if a framework is good or even just tell if it is popular or not. On the other hand, it costs so much delay / trouble if you choose a framework which does not fit well. But I think besides the total number of stars, the recent change on the number of star is also a good metric because an older repo tends to have more stars since it accumulated so much in the history. I actually created a website to track monthly change of stars for some frameworks I am interested in.

Answer 6

According to GitHub, star is used to bookmark, show appreciation, and for rankings.

According to research:

GitHub developers star repositories mainly to show appreciation to the projects (52.5%), to bookmark projects for later retrieval (51.1%), and because they used or are using the projects (36.7%).

The star button was born because once upon a time, people use fork to bookmark repos.
Why does GitHub call it "star" instead of "bookmark":

I created a web app that lists the top 1000 GitHub repositories and shows the total thumbs up 👍 of their top 5 closed PR/issues of the last 12 months. Determining if a repo is useful is tricky because usefulness is subjective and there's an overlap of "star to bookmark" and "star because I use this". So it's better to just check the repos individually and if you think it's of no use to you, blacklist it.