I'm going to answer your question from the perspective: what is the risk to open sourcing the technology that your friend's business has developed. I try to tackle your more specific question at the end.
TL,DR If your friend's company is built around the technology the company would open source, it's a real risk that their company would not be financially viable if the make the code available as open source, but it's unlikely to be as the result of a competing company forking their code.
If you venture outside the GPL-only ecosystem, I classify companies that support major open source projects in two categories:
- The open source project is an important part of their infrastructure but not central to their business (think Microsoft with .NET Core or Facebook with React)
- The open source project is the central to their business (think Cassandra and DataStax)
Within each category there are different ways to try to monetize the open source, but I think this high level view is sufficient for this question.
If your friend's company finds itself in the first category, putting the code out under an open source library is minimal risk to them in terms of other companies adopting it and using it to compete against them. For example, having access to React would not immediately let another company recreate Facebook's business and compete against them.
If your friend's company finds itself in the second category, they need to more carefully think about their business model. There are two major risks here:
- The vast majority of people and companies that use the open source project will do so without paying for it. The business must be sustainable based on income from the minority that chooses to pay. If the technology is in a niche business area where the pool of potential users is small, this may be unsustainable.
- If the business is too closely linked to the technology (e.g. all they do is provide consulting services to enhance the technology and help people install it), it would be possible for another business to provide the same service and do it better, whether that is with the same or with competing technology.
Your question is really about the second risk here, but the first risk will be the bigger one. Unless the company has convinced itself that there are enough users that are actually willing to pay for the use of the technology (or services associated with it, etc), then the company will not be viable if they make it open source.
How big a risk the second one is really depends on how unique and complex the technology is and how big the market for it is. Forking an open source project is kind of like any software maintenance work: there is a body of code that costs time and money to understand and improv. If the potential reward for doing this is not greater than the cost and time to do it, then it's not worthwhile.
If your friend's company's codebase is not that big and the market for it is not that large, it's probably not worth the effort for a competitor to fork the code and compete against your friend's company. If the codebase is not that big and the market is huge, then the risk is potentially real, but, because the codebase is not that big, it's equally likely a competitor would come in with their own implementation.
The most likely scenario for head-to-head competition based on a fork is large codebases with large markets. A ground up rewrite would be painful so a competitor would want to start with something it can already monetize. Companies built around Linux distributions as examples of this. StarOffice / OpenOffice / LibreOffice is another example.