Help:Accessing Cloud VPS instances
This page explains how to gain access to this wiki and access instances in the Wikimedia Cloud Services environment. This includes Toolforge and Cloud VPS. For example to ssh into one to run terminal commands.
The same key and password also let you access our git repositories located at https://gerrit.wikimedia.org/.
Prerequisites
- set up an SSH key if you do not already have one.
- Create a Wikimedia developer account on this wiki (Wikitech) and log in to this wiki.
- After logging in, upload your public ssh key to Wikitech.
- Also log in to gerrit using the same username and password. Click your name in the upper-right corner, choose Settings > SSH Public Keys, and upload your public key there as well.
Basic instructions for what to do next: Git workflow, Getting Started.
Accessing public and private Cloud VPS instances
Most Cloud VPS instances are not visible on the internet, so a hostname like host.project.eqiad.wmflabs means nothing to your local computer. To access them, you must proxy through another machine that is visible to the Internet and knows about the Cloud VPS internal network; we name these "bastion" instances. Everyone who is in a project has access to the bastion machines.
Almost everyone should proxy through primary.bastion.wmflabs.org. Members of the Wikimedia operations team must use restricted.bastion.wmflabs.org instead.
Accessing Toolforge instances
Accessing Toolforge instances can be done using a command like: ssh -i private_key user@login.tools.wmflabs.org where you replace user with your username on Toolforge, and you replace private_key with the path to your private key file.
After logging onto Toolforge, most actions should be conducted using a Tools project account. To do that, use the become command. For instance, if your tool is called my_project, after logging onto the Toolforge servers, run become my_project. This will ensure you have proper read and write access to files belonging to your tool. You can also use commands like jsub to submit jobs, etc.
Accessing instances with ProxyJump ssh option (recommended)
Note that this method should be available on any modern Unix-derived operating system environment (Linux, MacOS, *BSD, Windows Subsystem for Linux 1 or 2, possibly Cygwin, etc.) as well as on Windows 10 after the April 2018 release (before that the feature was in beta on Windows). If you are using the native SSH in Windows, you can run the ssh command from a cmd or Powershell prompt, and you would configure your .ssh/config file just like below in your Windows $HOME (and the OpenSSH agent can be enabled in services.msc as it is disabled by default).
Connecting to a typical Cloud VPS instance via a bastion host requires telling ssh to use the bastion a jump host with the ProxyJump directive:
$ ssh -J <your-shell-name>@primary.bastion.wmflabs.org <your-shell-name>@<your-instance>.<your-project>.eqiad.wmflabs
Typing all of that over and over again will be annoying, so it is common to add configuration to your $HOME/.ssh/config file to instruct ssh to always use primary.bastion.wmflabs.org as a jump host when connecting to wmflabs instances:
Host *.wmflabs
User <your-shell-name>
ProxyJump primary.bastion.wmflabs.org:22
Host primary.bastion.wmflabs.org
User <your-shell-name>
More details on how and why this works can be found in the Proxies and Jump Hosts section of the OpenSSH Cookbook.
Before OpenSSH 7.3, released in August 2016, the ProxyJump directive is not available. If this applies to you, you can use the ProxyCommand directive instead:
Host *.wmflabs
ProxyCommand ssh -a -W %h:%p <your-shell-name>@primary.bastion.wmflabs.org
User <your-shell-name>
Now you should be able to run the following directly from your local computer, substituting the instance and project names as appropriate:
ssh your-instance.your-project.eqiad.wmflabs
Improving ssh encryption protection
We recommend that users of OpenSSH 6.4 and newer configure your ssh client to only use strong ciphers when connecting to our servers:
Host *.wmnet *.wmflabs *.wikimedia.org !gerrit.wikimedia.org !git-ssh.wikimedia.org
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
ForwardAgent no
IdentitiesOnly yes
If you run an older version of ssh, you can either update to a more recent version or modify the configuration to use:
Host *.wmnet *.wmflabs *.wikimedia.org !gerrit.wikimedia.org !git-ssh.wikimedia.org
Ciphers aes256-ctr,aes192-ctr,aes128-ctr
KexAlgorithms diffie-hellman-group-exchange-sha256
ForwardAgent no
IdentitiesOnly yes
Accessing instances with a graphical file manager
You can directly connect to your Cloud VPS instance (through the bastion) via ssh with a graphical file manager.
- Windows: PuTTY and WinSCP.
- Linux (Gnome/KDE), the setup described above (Using ProxyCommand ssh option) is a prerequisite. Afterwards, you can connect like so:
- Gnome: In Nautilus, File -> Connect to server -> Type SSH, enter the instance's host name your-instance.eqiad.wmflabs and connect. In Nautilus version 3.6 and newer, use "Connect to server" and enter ssh://your-instance.eqiad.wmflabs as the address.
- Mac: Using sshfs you can mount a folder on labs to an (empty, existing) local folder for browsing with your favourite file browser. sshfs can be installed e.g. from the osxfuse project or from macports (port install sshfs). To write to folders where you have only group permissions, such as projects, you can specify defer_permissions:
sshfs -o defer_permissions your-instance.your-project.eqiad.wmflabs someLocalFolder
Accessing instances using agent forwarding
If configuring ProxyCommand doesn't work,
you can use an SSH agent, and forward that agent when connecting to the bastion instance so that from it you can ssh to particular Cloud VPS instances. For example:
- On your local system:
eval `ssh-agent` ssh-add ~/.ssh/your_key_file # (for instance: ~/.ssh/id_rsa ) ssh -A your-shell-name@primary.bastion.wmflabs.org
- Then on bastion:
ssh your-instance.your-project.eqiad.wmflabs
The above would let you ssh into your instance; note that FQDN isn't completely necessary; you can also just use ssh your-instance from within bastion.
If you already use ssh or an ssh agent to connect to other hosts, you should consider using a separate agent for your Cloud VPS key to avoid exposing your other keys to the Cloud VPS hosts. Also (depending on your operating system) you may want to purge keys that are automatically loaded using ssh-add -d after creating the agent. This article on ssh-agent may be helpful in that regard.
See Help:Putty for using PuTTY on Windows.
Accessing services using port forwarding
If your instance doesn't have a public IP address, but you need to access a service on it, you can use port forwarding. For instance, here's how you would access a web service:
On your local system
requires port 80 and 8080 to be part of the security group rules for the instance (managed via Horizon)
ssh username@primary.bastion.wmflabs.org -L 8080:your-instance:80 # Using agent forwarding to connect to bastion ssh username@primary.bastion.wmflabs.org -L 8080:your-instance:80 -A
Accessing web services with an http proxy
The easiest way to expose a web server to the public is via an HTTP proxy. This will provide public access to selected ports (generally just port 80) on an instance.
Accessing web services using a SOCKS proxy
If you need web access to an instance but do not wish it to be open to the public, a SOCKS proxy will allow you to set up access that's limited to your ssh bastion login.
Setting up the proxy
A SOCKS proxy makes it possible to connect directly to a Cloud VPS instance from your browser's location field, so browsing to http://your-instance.eqiad.wmflabs:8080 will work.
- On your local system:
ssh username@primary.bastion.wmflabs.org -D 8080 - Configure your computer to use localhost:8080 as a SOCKS proxy. You can configure this in a few different ways (whichever fits you best):
- On a Mac, open System Preferences > Network > Advanced > Proxies. Tick "Socks Proxy" and enter localhost and 8080. After saving and applying this you can browse to http://your-instance.eqiad.wmflabs:8080 in the browser.
- If you use foxy proxy (a Firefox, IE or Chrome add-on), you can use the pre-defined URL pattern defined here. After adding the URL patterns, you'll also need to edit the defined proxy to specify localhost as the host and 8080 as the port.
Troubleshooting the proxy
Test to see if the proxy is working:
- load http://whatismyipaddress.com/. If you see your own IP address, your browser is not sending traffic through the proxy. If you see 208.80.153.207, the browse is correctly sending traffic through the proxy.
- If you see a Server Not Found error when you browse to http://your-instance.eqiad.wmflabs you may need to tell your browser to use remote DNS instead of local.
- To do this on Firefox, point your browser at about:config and search for network.proxy.socks_remote_dns. If it is set to false double click on the row to set it to true.
Troubleshooting
In general, adding ssh option -v, -vv, or -vvv may help identify possible issues.
#when using Agent Forwarding ssh -Av username@primary.bastion.wmflabs.org #when using ProxyCommand ssh -v your-instance.your-project.eqiad.wmflabs
Into Bastion
Permission denied (publickey)
- Make sure you have uploaded the correct ssh key to your preferences
- Use lowercase letters for your username
- Your ssh user name is your instance shell account name name (see User Profile > Basic Information in your WikiTech account's Preferences page). It is not necessarily the same as your account's username
Connection closed by remote host
- Make sure you have uploaded the correct ssh key to your preferences
- If you have access to other ssh servers, can you connect to them? If not, then there may be an issue with your ssh client.
- If you use Windows, is Pageant set up with correct keys and running?
- Contact a Wikimedia Cloud Services admin via #wikimedia-cloud connect IRC and let them know about your issue.
They can look into the following :- make sure your username is in 'bastion' project details in openstack-browser
- clear any cache settings or deny host settings that may also prevent you from ssh’ing into bastion.
Blocking connection on OS X with no error message
If you are running OS X and your ssh connection blocks without any error message (while pinging the server works), try
unset SSH_AUTH_SOCK
and then ssh again. This will unset the socket to ssh-agent.
Into your-instance
Permission denied (publickey)
- Make sure the instance build has actually completed.
Look in the console output for “Finished puppet run”, BEGIN SSH HOST KEY FINGERPRINTS and BEGIN SSH HOST KEY KEYS output. - If using Using agent forwarding, make sure you ssh’d into Bastion with the -A option.
Banners
If you log into any instance apart from bastion, ssh will display:
If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances
This message cannot be suppressed, and you need to manually filter it if you want to process the output.
Rights
Anonymous users
Sign up for a Wikimedia developer account here: Request account (you will be asked to enter the new account's information)
Logged-in users
After creating an account, you can:
Once you add a key, you'll be able to log into the instance of any project you are a member of. Have a current project member, or an admin user add you to a project.
To create instances within a project, you'll need to have a projectadmin on the project, or an admin add you to the projectadmin role for that project.
After logging in, you can also access Gerrit; if you wish to do git checkouts of the puppet repositories, you'll need to log into Gerrit, and add your SSH key there as well.
Project Members
If you are a member of a project (to check, see https://tools.wmflabs.org/openstack-browser/project/projectName and expand the list of Users) then you can:
- ssh to project instances
Admins
Project Admins
If you are a Project Admin, you can:
- Allocate and associate public IP addresses, and add public DNS entries to instances
- Manage security groups and rules.
- Create or manage instances.
- Manage available puppet classes and variables.
- Manage sudo policies.
- Manage project and project role membership.
Cloud Admins
In addition to all actions that projectadmins can do, cloudadmins can:
Wiki Admin
If you are a wiki admin, you can:
Puppet
Cloud VPS instances are configured from the operations/puppet repository and branch 'production'.
To trigger a puppet update, run: sudo -i puppet agent --test --verbose
The cron job that updates the master puppet repo in the Cloud VPS cluster runs every minute. So your change got merged in, you will have to wait a bit.
Access FAQ
- Q: How do I get access to Cloud VPS?
- A: See the Getting Started guide.
- Q: I just want git access; I don't care about wikitech or Cloud VPS. How do I get Git access?
- A: Wikitech, Gerrit, Phabricator, Cloud VPS, Toolforge, and other services all use the same account backend for authentication. See the Getting Started guide to create your Wikimedia developer account.
- Q: I was added to a group that gives me access to something in git, but it isn't working, what's wrong?
- A: Once you have been added to the group, you need to log out of gerrit, then back in. Gerrit pulls its groups from LDAP, but caches them. Logging out, then back in re-synchronizes your groups, and thus clears the cache.
- Q: How do I add a new tool to Toolforge or migrate a tool from the toolserver to Toolforge?
- A: See the Toolforge sections in the Getting Started guide. To migrate a tool, or for general insight, read these instructions from User:Magnus Manske.