Forever free: building non-profit online services in a sustainable way

In the past decade, we have seen a big switch from client run software to online services. Some services such as scientific data repositories were a natural fit for a centralized online implementation (but one day we can see a distributed versions of those). Others, such as Science-as-a-Service platforms, were more convenient and scalable versions of the client/desktop based software. One thing is certain - online platforms, available 24/7 via a web browser have proven to be very convenient in a range of tasks such as communication, sharing data, and data processing.

Non-profit sector (such as projects funded by scientific grants) has also entered this domain. There are countless examples where modern web technologies based on centralized services can benefit scientists and general public even if the service they provide is not part of a commercial operation. This is especially true due to increased trend to share data and materials in science. Those outputs need to be stored and made available in centralized repositories. Additionally, data processing service can be provided to incentivize sharing and increase the value and accessibility of deposited data (this is the modus operandi of OpenNeuro.org - a platform for sharing and analysis of human brain imaging data).

Online services are great - I am basically addicted to Gmail web client, not mentioning Google Docs. Even though we have seen many successful non-profit/academic client based toolboxes or libraries, developing and maintaining online services over a long period of time comes with a host of other challenges. Funding in non-profit sector can be finicky - it’s easy to put the active development of a software package aside for a year while funding is not there. How can an online platform be designed to survive such dry patches? How can one build a thriving open source community around a hosted service?

Below are some ideas on how to design and maintain non-profit hosted software projects in a way that increases their chance to survive and thrive in the long term.

1. Develop the “bus accident” mindset

Many academic or non-profit projects revolve around a single individual. Even if there are more people involved, many times the passion of one person drives the project forward. If you are reading this, it's probably you I am talking about. When running an online project it’s useful to ask a hypothetical question “What would happen to my project if tomorrow I will get hit by a bus?”. In the spur of the moment, high on getting things done attitude it’s easy to forget how much can rely on one person. Having the “bus accident” mindset will allow you to start thinking about redundancies, delegation, backup and long term plans. Many of the topics covered below come from this mindset.

2. Choose your framework wisely

One of the most important decisions you will have to make when building an online platform is which framework to use. There are several factors to take into account: size of the development community (especially for third party extensions), maturity (how long has it been around), programming language (important in the context of building the developer community for your project), and the number and size of existing projects using the framework. You might be tempted to opt for some hipster cutting edge new framework written in an obscure language you always wanted to learn. Such choice might bite you back in the future when the chosen framework will fail to attract a large community and none of your colleagues will have time to learn a new programming language to contribute to your project.

3. Build a developer community

Some aspects of software project building online services are no different from those building client side applications or frameworks. For example, both benefit tremendously from a healthy open source community. The premise is very simple the more people you get involved the better your service will become. Furthermore, responsibilities can be spread across more people removing the “single point of failure”. Building a community for your software project is not as simple as putting it on GitHub though. It involves writing guides for new contributors, setting additional communication channels (Gitter or Slack), implementing a code of conduct, curating a set of easy issues to tackle for beginners, crediting all of the contributors appropriately etc. You can read how to achieve this for your project in this excellent guide. Soon your project will not only be better integrated with the community you serve but also have a small army of volunteers to help you maintain it.

4. Automate all the things

Your time is precious. While the funding for the project is solid it might not seem to be problematic that testing, code review, and deployment are manual operations. However, when the project hits a dry patch and responsibilities will have to be spread across more people digging up all the manual procedures and implementing them will cost time and slow things down. This is especially true if most contributors work on the project part time. One way to improve the situation is to automate as much as possible. Testing can be automated via Continuous Integration services such as CircleCI or Travis (both free for open source projects). The same services can also help with automatic deployment when a new release is made. Even code review - an intrinsically manual task - can be partially automated using static code analysis services such as CodeClimate (also free for open source projects). Future you will thank you for the times spend setting up those automations because it will allow you to run the project without full time involvement.

5. Make debugging collaborative

Another challenge of collaborative software projects that work on a centralized online platform is debugging. When debugging a library or a framework things are relatively straightforward: you get some example inputs/data and run things locally to try to replicate the bug. Things are a bit harder with online platforms because they feature more moving parts - databases, asynchronous workers, different versions of web browser etc. Replicating a bug user reports can be challenging without access to all logs on the server. For security concerns, this access cannot be shared publicly. However, there is a growing family of services that can be integrated with your platform that crawl error logs, group them together with contextual information (Which user? Which browser? etc.). This not only helps to debug online platforms but also allows to make this process collaborative, since you can give access to those logs to your contributors. Two companies providing such services worth looking into are OpBeat and Sentry (both free for open source projects).

6. Be ready to move

Wherever your platform is currently hosted: AWS, GCP, Azure, your institution, or private hardware this might not be an optimal choice in the future. Maybe a competitor will provide cheaper rates. Maybe your institution will step in and provide subsidized space. Maybe during a dry patch of funding, you will have to move to hosting it on your own hardware (I know of a major online platform in human neuroimaging that is hosted on a computer sitting under the desk of it’s the main developer - surprisingly cost effective solution!). The bottom line is that you almost certainly will have to move. This process can be complicated due to different software dependencies of your platform. One way to ease this pain is to use Docker for capturing all of the dependencies and docker-compose for orchestrating multiple services (databases, workers, etc. An added bonus is that the same technology will allow your contributors to quickly set up a dev version of the platform on their machines.

7. Make it modular

Many non-profit online services have different features that have different costs related to storage and compute. For example NeuroVault.org repository of human brain maps offers image based search feature. This feature is not essential to the main mission of the portal (sharing data) and it does come with extra cost necessary to cover extra database space and similarity computations. Similar features can be found in other services.
When funding is uncertain it is beneficial to design the system so certain features could be turned off during periods with fewer financial resources. This way you can quickly reduce the day to day cost of running the service until more resources can be found.

8. Make it scalable

Sometimes things are less “black and white”. Certain features could be scaled up or down - for example in terms of how long users will have to wait to receive results of their analysis or how much storage they can use. This is another place where you can design your system to adapt to changing funding landscape. This can mean opting for an auto scaling fleet of asynchronous workers (that work across machines) instead of building one multithreaded process that requires a large expensive machine. The work put into designing a scalable architecture will pay off later.

9. Back up all the things

Backup with a third party is an essential part of long term preservation of non-profit data repository. One should aim at a situation where in case all developers of the project disappear an interested party would be able to revive the repository using code from GitHub (with nice deployment routines thanks to being “ready to move”) and data backups from a public space? But doesn’t space cost money? In general, yes, but one can use subsidized data repositories such as Open Science Framework, Harvard Dataverse or Figshare for free. Additionally, some institutions provide long term storage for free for projects affiliated with them.

10. Reuse instead of implementing from scratch

There are many components of online platforms that you might be tempted to implement (and more importantly maintain) yourself but could be replaced by third party services. Why deal with recovering lost passwords when social logins (Google, Facebook, WeChat etc.) could be used for free instead? Need the ability to comment on content - embed Disqus. Need to deal with sending and receiving emails? The free quota of Mailgun is probably good enough for most non-profit projects. This will not only save time implementing this functionality but more importantly, it will offset the cost of maintaining it.

11. Consider monetization

When grant money runs out you might consider asking your users to help you out to pay the AWS bills. There are several ways to do this. You can ask your users for donations (this is how Wikipedia runs!) via a service such as Fundly. You can also enroll your platform in the Flattr program which enables users to dedicate a set amount of money monthly for donations that are later split between services they use (as tracked by Flattr).

Adding ads is another option, but it may put some of your users off (nonetheless something worth exploring). An interesting alternative to showing annoying ads (which most people block anyway) is to ask your users if you can use their computer to mine cryptocurrency. This can be implemented via CoinHive. Just remember to use the version that asks users for explicit permission. This scheme only makes sense if users spend a lot of time on your website.

Monetization even though attractive might require you to formalize your project into a non-profit organization for tax purposes. Something to keep in mind before taking this route.

12. Consider industrial partnerships

Another option to inject stability into your service is partnering with the industry. The way to look at such partnership is an exchange of good PR for resources. Depending what your platform is doing there might be companies that would be interested in helping you out in exchange for exposure to your user base. Big companies such as Amazon and Google provide a version of such partnerships via their Open Datasets programs. This only helps data repositories (compute is not subsidized) but is nonetheless a very attractive way to help maintain your platform.

Online services such as data repositories or Science-as-a-Service platforms can be very valuable and even essential for some types of scientific questions. However, relying on short term grants it’s hard to maintain them over a long period of time. I hope the above pointers can help make your portal more sustainable - if I missed anything please let me know in the comments below. Stay strong, keep building cool things, and most importantly - don’t get hit by a bus!