Help:Cloud Services introduction: Difference between revisions

From Wikitech
Browse history interactively
← Older edit
Content deleted Content added
VisualWikitext
m Reverted edits by Mayankbadhani (talk) to last revision by SRodlund
Tag: Rollback
m Reverted edits by NoobThreePointOh (talk) to last revision by Majavah
Tag: Rollback
 
(78 intermediate revisions by 28 users not shown)
Line 1: Line 1:
[[File:What is Cloud Services? poster.pdf|alt=A Poster showing some Wikimedia Cloud services statistics and services|thumb|upright=1.3|Poster-format overview]]
{{notice | wmflabs.org and wmcloud.org redirect here. You might be looking for the [[:Category:Toolforge tools|Lists of Toolforge tools]] or the [[toolforge:openstack-browser/project/|List of Cloud VPS projects]].}}
'''Wikimedia Cloud Services (WMCS)''' provides tools, services, and support for technical collaborators who want to contribute to Wikimedia software projects. Use Cloud Services to host your software tools for the [[meta:Special:MyLanguage/Wikimedia movement|Wikimedia movement]], without charge.


[[File:What is Cloud Services? poster.pdf|thumb|upright=1.3|Poster-format overview]]
== What can you do with Cloud Services? ==


=== Host tools on Wikimedia servers ===
== WMCS project overview ==
{{anchor|Toolforge|reason=Old section name.}}


Tools and bots make it easier to edit and maintain Wikimedia projects. For developers who support Wikimedia projects by developing tools and bots, '''[[Help:Toolforge | Toolforge]]''' provides the following features:
===What is Wikimedia Cloud Services (WMCS)?===
* Free, reliable, and scalable shared hosting, including web servers, databases and other data storage
* A distributed job processing system
* Support for multiple users to collaboratively maintain and manage tools


To use Toolforge you need:
'''Wikimedia Cloud Services''' ('''WMCS''') provides tools, services, and support for technical collaborators who want to contribute to Wikimedia software projects. WMCS is a computing ecosystem built on [[W:OpenStack|OpenStack]], [[W:Oracle Grid Engine|GridEngine]], and [[W:Kubernetes|Kubernetes]].
* Some programming knowledge
* An understanding of Unix command line


To get started, visit [[Help:Toolforge]]. Or, [https://developer.wikimedia.org/build-tools/ learn more about creating bots].
WMCS products and services are available for use by anyone connected with the [[meta:Wikimedia movement|Wikimedia movement]] without charge. Support and administration of the WMCS resources is provided by a [[Mw:Wikimedia Cloud Services team|Wikimedia Foundation Cloud Services team]] and [[wmf:Volunteer_opportunities|Wikimedia movement volunteers]]. We maintain a [[Help:Glossary|Glossary]] of related terminology.


=== Run scripts and visualize data ===
🎬 '''Video''': [https://media.ccc.de/v/36c3-77-wikimedia-cloud-services-introduction Wikimedia Cloud Services introduction]
{{anchor|PAWS|reason=Old section name.}}
'''[[PAWS]]''' is a Jupyter notebook installation hosted by Wikimedia. PAWS notebooks can be used for creating tutorials, running live code, creating data visualizations, running basic bots, and more.


A single PAWS notebook is maintained by a single user, but they can be downloaded and forked by other users. To use PAWS you need only a Wikimedia login and a web browser. Knowledge of Python is helpful, but not required.
📣 '''Slides''': [[commons:File:Introduction_to_Wikimedia_Cloud_Services_-_Wikimania_Hackathon_2019_Stockholm_Sweden.pdf|An introduction to Cloud Services presentation from Wikimania Stockholm 2019]]


=== Administer servers for software development ===
=== WMCS history ===
{{anchor|Cloud VPS|reason=Old section name.}}
Open source software projects help the Wikimedia movement by improving core infrastructure (like MediaWiki), powering research and analytics, and supporting Wikimedia operations and software development. For advanced projects that aren't viable in the Toolforge environment, '''[[Help:Cloud_VPS | Cloud VPS]]''' (Virtual Private Server) provides the following features:
* Free cloud computing environment, powered by [[:w:OpenStack|OpenStack]]
* Collaboratively-owned collections of virtual private servers, storage, firewall, and HTTPS proxy resources to projects
* Access to a variety of data services
* Freedom to install packages not provided by Debian or the Wikimedia Foundation
To use Cloud VPS, you need:
* An open source project that isn't viable in the Toolforge environment, or can't be accomplished using other WMCS offerings
* One or more active project maintainers who meet basic requirements
* Advanced programming knowledge
* Advanced experience with Unix command line
* The ability to administer your own servers and manage your project's applications, data, runtime, middleware, and operating systems


To get started, visit [[Help:Cloud VPS]].
From 2011 until early 2017, WMCS was known as "Wikimedia Labs." The term 'Labs' was used to refer to a number of different [[Labs labs labs|components]], and clarification was required. In 2017, the project was reorganized. The former Wikimedia Foundation Labs team and the Tool Labs Support team joined together to create the Wikimedia Cloud Services team.


==== What is the difference between Cloud VPS and Toolforge? ====
{{anchor|WMCS Products}}


Cloud VPS is an [[:w:Cloud_computing#Infrastructure_as_a_service_.28IaaS.29|Infrastructure as a service (IaaS)]] solution. It provides virtual machines, storage, firewall, and HTTPS proxy resources to projects. The members of each individual project are responsible for managing applications, data, runtime, middleware, and operating systems themselves.
== WMCS products and services ==


Toolforge is a [[:w:Cloud_computing#Platform_as_a_service_.28PaaS.29|Platform as a service (PaaS)]] solution. It provides [[Help:Toolforge/Web|web servers]], [[Help:Toolforge/Database|databases]], and a [[Help:Toolforge/Grid|distributed job processing system]] as managed services for tool maintainers.
{{anchor|What product should I use?}}
{| class="wikitable sortable"
|+ WMCS Products
! Service
! Product
! Description
! Use
! Support Level
|-
| [[:en:Virtual_private_server|VPS]]
| [[Portal:Cloud VPS|Cloud VPS]]
| Provides collaboratively owned collections of virtual private servers where users develop and maintain software projects that help the Wikimedia movement.
| Use this to run full virtual instances.
| You are willing to administer instances on your own. We can provide quota to do so.  
|-
| [[:en:Platform as a service|PaaS]]
| [[Portal:Toolforge|Toolforge]]
| Provides a shared hosting/platform-as-a-service environment for running bots, webservices, scheduled jobs, and data analysis.
| Run a specific webservice, scheduled job, or perform analysis.
| You do not want to or are not able to manage a full virtual environment.
|-
| [[:en:Data_as_a_service|DaaS]]
| [[Portal:Data Services|Data Services]]
| A collection of products including private-information-redacted copies of Wikimedia's production wiki databases and access to [[Dumps.wikimedia.org|Wikimedia Dumps]].
| Create replicas of the production databases and other data for analysis and experimentation.  
| The [[quarry:|Quarry service]] provides database access via a web interface. Some DaaS resources may need to be requested for specific VPS projects.
|}


=== Renaming of products and services ===
=== Access databases and data dumps ===
{{anchor|Data Services|reason=Old section name.}}
We are in the process of changing the [[phab:phame/post/view/59/labs_and_tool_labs_being_renamed/| language and branding]] of the products and services we offer. You may find some outdated titles and names in WMCS documentation. Edits are welcome!
{{Note|What is a wiki replica? What's in the dumps? Learn the basics of Wikimedia open data and how to access all available datasets at [[meta:Research:Data|Research:Data]].}}


==== Access wiki databases for tool development====
== Participating with WMCS ==


Tools and software hosted on Toolforge and Cloud VPS can directly access public data dumps and production wiki replicas.
=== Sign up for services ===
* Learn about [[Help:Toolforge/Database | accessing wiki replica databases from a tool account]].
* Learn about [[Help:Shared_storage | accessing dumps through shared storage services for Cloud VPS and Toolforge]].


==== Query wiki replicas and dumps in a browser ====
To access and contribute to Cloud Services projects and tools, you will need the following accounts:


* '''[[PAWS]]''' provides a Jupyter notebook environment you can use to query wiki replicas and dumps, create interactive graphs, and use APIs for analysis.
*'''Wikimedia account''' - this account is the single user login or SUL account you use to contribute to Wikipedia and its sister projects.
* '''[[Superset]]''' and '''[https://quarry.wmcloud.org/ Quarry]''' are web interfaces for querying live replica SQL databases of public Wikimedia wikis.
*[[Help:Create_a_Wikimedia_developer_account|'''Wikimedia developer account''']] - this account is used log into this wiki, Toolforge, Cloud VPS, Gerrit and other protected Wikimedia Services.
To use Superset or Quarry you need only a Wikimedia login and a web browser, but you should have a basic understanding of SQL. Learn about [[Help:MySQL_queries|SQL queries]].
* '''[[mw:Gerrit|Gerrit]]'''- our code review system; where our repositories (repos) live. Note that while [[mw:Github|Github]] contains many of our public repos, you can only make pull requests for Cloud Services projects via Gerrit. Other Wiki projects may use Github exclusively.
* '''[[mw:Phabricator|Phabricator]]'''- our project management system; for opening tickets, suggesting features, and talking about our plans for the next quarter.
* '''[[mw:MediaWiki on IRC|IRC]]''' - live chat channels. We have several channels related to our cloud servers, but the main channel is #wikimedia-cloud. Deployment also frequently uses #wikimedia-serviceops.
* The ability to [[Help:Access|access instances]] in the WMCS environment.


== Which service is right for you? ==
===Review the terms and conditions===


{| class="wikitable"
Second, make sure to review and agree to our terms and conditions. [[Help:Terminology|Account holders]] who plan to use WMCS resources and products must read and agree to the following:
|+
! rowspan="2" |Activity / Needs
!PAWS
!Superset
!Toolforge
!Cloud VPS
|-
|Data as a service
|Data as a service
|Platform as a service
|Infrastructure as a service
|-
|Write scripts and visualize data
|✔
|
|
|
|-
|Write queries against wiki replica databases
|✔
|✔
|✔
|✔ via Toolforge
|-
|Access on wiki database dump files
|✔
|
|✔
|
|-
|Write and run bots
|✔
|
|✔
|✔ if not viable on Toolforge
|-
|Run web services
|
|
|✔
|✔ if not viable on Toolforge
|-
|Build tools to improve Wikimedia projects
|
|
|✔
|✔ if not viable on Toolforge
|-
|Schedule or run continuous jobs
|
|
|✔
|✔ if not viable on Toolforge
|-
|Need your own subdomain
|
|
|✔
|✔
|-
|Work with co-maintainers and co-admins
|
|
|✔
|✔
|-
|Install packages not provided by Debian or the Wikimedia Foundation
|
|
|
|✔
|-
|Administer your own virtual server
|
|
|
|✔
|-
!Platform / Environment
|web browser
|web browser
|terminal
|terminal
|-
!User knowledge
|curious—advanced
|curious—advanced
|intermediate—advanced
|advanced
|}


== Before you start ==
* [[Wikitech:Cloud Services Terms of use|Wikimedia Cloud Services Terms of Use]]
* [[mw:Code of Conduct|Code of Conduct for technical spaces]]
* [[mw:Wikimedia Labs/Agreement to disclosure of personally identifiable information|Agreement to disclosure of personally identifiable information]] (covers [[Help:Terminology|End-Users]]).


To use Cloud Services products, you must first [[Help:Create a Wikimedia developer account|create a Wikimedia account and a developer account]].
Please pay close attention to the following terms for '''Toolforge and Cloud VPS:'''

* Toolforge tools must be [[w:Open-source software|open source software]] licensed under an [http://opensource.org/licenses OSI approved license].
* Toolforge and Cloud VPS projects must not collect, store, or share private data or personally identifiable information, such as user names, passwords, or IP addresses, except when complying with the conditions listed in the [[Wikitech:Cloud Services Terms of use|Wikimedia Cloud Services Terms of Use]].

{{:Help:Cloud Services communication}}


== Learn more ==
== Learn more ==


* 🎬 Video: [https://media.ccc.de/v/36c3-77-wikimedia-cloud-services-introduction Wikimedia Cloud Services introduction] (2019)
* [[Help:FAQ | Cloud VPS and Toolforge At-A-Glance]]: This page provides a basic introduction to '''Cloud VPS''' and '''Toolforge'''.
* 📣 Slides: [[commons:File:Introduction_to_Wikimedia_Cloud_Services_-_Wikimania_Hackathon_2019_Stockholm_Sweden.pdf|An introduction to Cloud Services presentation]] (2019)
* [[Portal:Cloud_VPS | Cloud VPS Portal]]: Information about Cloud VPS and links to help and technical documentation.

* [[Portal:Toolforge | Toolforge Portal]]: Information about Toolforge and links to help and technical documentation.
{{:Help:Cloud Services communication}}
* [[Portal:Data_Services | Data Services Portal]]: Information about Data Services and links to help and technical documentation.


[[Category:Overviews]]
[[Category:Cloud Services]]
[[Category:Cloud Services]]

Latest revision as of 21:35, 3 February 2024

A Poster showing some Wikimedia Cloud services statistics and services
Poster-format overview

Wikimedia Cloud Services (WMCS) provides tools, services, and support for technical collaborators who want to contribute to Wikimedia software projects. Use Cloud Services to host your software tools for the Wikimedia movement, without charge.

What can you do with Cloud Services?

Host tools on Wikimedia servers

Tools and bots make it easier to edit and maintain Wikimedia projects. For developers who support Wikimedia projects by developing tools and bots, Toolforge provides the following features:

  • Free, reliable, and scalable shared hosting, including web servers, databases and other data storage
  • A distributed job processing system
  • Support for multiple users to collaboratively maintain and manage tools

To use Toolforge you need:

  • Some programming knowledge
  • An understanding of Unix command line

To get started, visit Help:Toolforge. Or, learn more about creating bots.

Run scripts and visualize data

PAWS is a Jupyter notebook installation hosted by Wikimedia. PAWS notebooks can be used for creating tutorials, running live code, creating data visualizations, running basic bots, and more.

A single PAWS notebook is maintained by a single user, but they can be downloaded and forked by other users. To use PAWS you need only a Wikimedia login and a web browser. Knowledge of Python is helpful, but not required.

Administer servers for software development

Open source software projects help the Wikimedia movement by improving core infrastructure (like MediaWiki), powering research and analytics, and supporting Wikimedia operations and software development. For advanced projects that aren't viable in the Toolforge environment, Cloud VPS (Virtual Private Server) provides the following features:

  • Free cloud computing environment, powered by OpenStack
  • Collaboratively-owned collections of virtual private servers, storage, firewall, and HTTPS proxy resources to projects
  • Access to a variety of data services
  • Freedom to install packages not provided by Debian or the Wikimedia Foundation

To use Cloud VPS, you need:

  • An open source project that isn't viable in the Toolforge environment, or can't be accomplished using other WMCS offerings
  • One or more active project maintainers who meet basic requirements
  • Advanced programming knowledge
  • Advanced experience with Unix command line
  • The ability to administer your own servers and manage your project's applications, data, runtime, middleware, and operating systems

To get started, visit Help:Cloud VPS.

What is the difference between Cloud VPS and Toolforge?

Cloud VPS is an Infrastructure as a service (IaaS) solution. It provides virtual machines, storage, firewall, and HTTPS proxy resources to projects. The members of each individual project are responsible for managing applications, data, runtime, middleware, and operating systems themselves.

Toolforge is a Platform as a service (PaaS) solution. It provides web servers, databases, and a distributed job processing system as managed services for tool maintainers.

Access databases and data dumps

What is a wiki replica? What's in the dumps? Learn the basics of Wikimedia open data and how to access all available datasets at Research:Data.

Access wiki databases for tool development

Tools and software hosted on Toolforge and Cloud VPS can directly access public data dumps and production wiki replicas.

Query wiki replicas and dumps in a browser

  • PAWS provides a Jupyter notebook environment you can use to query wiki replicas and dumps, create interactive graphs, and use APIs for analysis.
  • Superset and Quarry are web interfaces for querying live replica SQL databases of public Wikimedia wikis.

To use Superset or Quarry you need only a Wikimedia login and a web browser, but you should have a basic understanding of SQL. Learn about SQL queries.

Which service is right for you?

Activity / Needs PAWS Superset Toolforge Cloud VPS
Data as a service Data as a service Platform as a service Infrastructure as a service
Write scripts and visualize data ✔
Write queries against wiki replica databases ✔ ✔ ✔ ✔ via Toolforge
Access on wiki database dump files ✔ ✔
Write and run bots ✔ ✔ ✔ if not viable on Toolforge
Run web services ✔ ✔ if not viable on Toolforge
Build tools to improve Wikimedia projects ✔ ✔ if not viable on Toolforge
Schedule or run continuous jobs ✔ ✔ if not viable on Toolforge
Need your own subdomain ✔ ✔
Work with co-maintainers and co-admins ✔ ✔
Install packages not provided by Debian or the Wikimedia Foundation ✔
Administer your own virtual server ✔
Platform / Environment web browser web browser terminal terminal
User knowledge curious—advanced curious—advanced intermediate—advanced advanced

Before you start

To use Cloud Services products, you must first create a Wikimedia account and a developer account.

Learn more

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support
Stay aware of critical changes and plans
Track work tasks and report bugs

Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself

Read stories and WMCS blog posts

Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Help:Cloud_Services_introduction&oldid=2145397"