Help:Cloud Services introduction
Wikimedia Cloud Services (WMCS) provides tools, services, and support for technical collaborators who want to contribute in Wikimedia software projects. Use Cloud Services to host your software tools for the Wikimedia movement, without charge.
Service concepts
Superset and PAWS empower technically curious to advanced users to query wiki replicas and create scripts, tutorials, and data visualizations to analyze and improve Wikimedia projects.
See #Data Services below.
Toolforge is for intermediate to advanced users working on tools, bots, webservices that support Wikimedia projects.
See #Toolforge below.
Cloud VPS is for advanced users who need to administer their own servers for Wikimedia operations and software development.
See #Cloud VPS below.
Are you unsure? Check #Which service is right for you? below.
Toolforge
Toolforge is one of the projects hosted by Wikimedia Cloud VPS. It is a shared hosting (platform as a service) environment for volunteers to develop and run tools, continuous bots, web services, scheduled jobs, and data analysis.
To use Toolforge you will need some programming knowledge, an understanding of Unix command line, and version control via Gerrit and Git.
Users of the Toolforge project create so-called "tool" accounts (technically service groups). These accounts allow one or more users to collaborate to manage the software source code, configuration, and jobs for that tool or bot.
The Toolforge administrators manage a pool of virtual servers which provide a shared project hosting environment that can be used by Toolforge users. These resources include web servers, databases and other data storage, and a distributed job processing system. These services provide a reliable and scalable hosting environment for volunteers to develop and operate their tools and bots.
For additional documentation and help with Toolforge, see Portal:Toolforge.
Cloud VPS
Cloud VPS (Virtual Private Server) is a cloud computing environment powered by OpenStack. It offers collaboratively owned collections of virtual private servers. You can use this infrastructure to create and maintain open source software projects that help the Wikimedia movement.
The environment includes access to a variety of data services. Cloud VPS allows developers and system administrators to try out improvements to Wikimedia infrastructure (including MediaWiki), power research and analytics, and host projects that are not viable in the Toolforge environment.
What is the difference between Cloud VPS and Toolforge?
Cloud VPS is an Infrastructure as a service (IaaS) solution. It provides virtual machines, storage, firewall, and HTTPS proxy resources to projects. The members of each individual project are responsible for managing applications, data, runtime, middleware, and operating systems themselves.
Toolforge is a Platform as a service (PaaS) solution. It provides web servers, databases and other data storage, and a distributed job processing system as managed services that can be used by tools and their maintainers.
Data Services
Data Services are a collection of products which provide access to copies of Wikimedia's production wiki databases (with private information redacted) and access to Wikimedia data dumps. Use data services to create replicas of the production databases and other data for analysis and experimentation.
There are also services to interact with data in a web browser: Superset and PAWS.
Superset
An instance of Superset is offered as a public querying interface for Wiki Replicas, a set of live replica SQL databases of public Wikimedia wikis. Superset offers a web interface for running queries against the Wiki Replicas. Making exploring the replicas relatively easy.
To use Superset you need only a Wikimedia login and a web browsers. A basic understanding of SQL is recommended. Learn about SQL queries.
PAWS
PAWS is a Jupyter notebook installation hosted by Wikimedia. PAWS notebooks can be used for creating tutorials, running live code, creating data visualizations, running bots using Pywikibot, and more.
A single PAWS notebook is maintained by a single user, but they can be downloaded and forked by other users.
To use PAWS you need only a Wikimedia login and a web browser. Knowledge of Python is helpful, but not required.
Which service is right for you?
| Activity / Needs | PAWS (DaaS) | Superset (DaaS) | Toolforge (PaaS) | Cloud VPS (IaaS) |
|---|---|---|---|---|
| Data as a service | Data as a service | Platform as a service | Infrastructure as a service | |
| Write documentation and create tutorials | ✔ | |||
| Browser based | ✔ | ✔ | ||
| Write queries against replica databases | ✔ | ✔ | ✔ | ✔ via Toolforge |
| Access on database dump files | ✔ | ✔ | ||
| Write and run bots | ✔ | ✔ easily | ✔ manually | |
| Run web services | ✔ easily | ✔ manually | ||
| Build tools to improve Wikimedia projects | ✔ easily | ✔ manually | ||
| Schedule or run continuous jobs | ✔ easily | ✔ manually | ||
| Terminal based | ✔ | ✔ | ||
| Need your own subdomain | ✔ | ✔ | ||
| Work with co-maintainers and co-admins | ✔ | ✔ | ||
| Install packages not provided by Debian or the Wikimedia Foundation | ✔ | |||
| Administer your own virtual server | ✔ | |||
| User knowledge | curious—advanced | curious—advanced | intermediate—advanced | advanced |
Get started
Set up your Toolforge or Cloud VPS projects by following the instructions on Help:Getting Started.
Communication and support
Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:
- Chat in real time in the IRC channel #wikimedia-cloud connect or the bridged Telegram group
- Discuss via email after you have subscribed to the cloud@ mailing list
- Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)
- Read the News wiki page
Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself
Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)
Technology stack
WMCS is a computing ecosystem built on OpenStack and Kubernetes.
Learn more
- Toolforge Portal — Information about Toolforge and links to help and technical documentation.
- Cloud VPS Portal — Information about Cloud VPS and links to help and technical documentation.
- Data Services Portal — Information about Data Services and links to help and technical documentation.
- See the glossary for detailed definitions of terms which are specific to Toolforge and Cloud VPS.
- 🎬 Video: Wikimedia Cloud Services introduction (2019)
- 📣 Slides: An introduction to Cloud Services presentation (2019)
Historical information
From 2011 until early 2017, Wikimedia Cloud Services was known as Wikimedia Labs. However, the term Labs was used for several different things.
Since 2017, the former Wikimedia Foundation Labs team and Tool Labs Support team merged into the Wikimedia Cloud Services team.