Help:Accessing Cloud VPS instances: Difference between revisions

From Wikitech
Browse history interactively
← Older editNewer edit →
Content deleted Content added
VisualWikitext
Ryan Lane (talk | contribs)
2,117 edits
→‎What you'll need: Add link to phab task for improvements to "What you'll need" section
(282 intermediate revisions by 91 users not shown)
Line 1: Line 1:
{{Cloud VPS nav}}
== Anonymous users ==


== Overview ==
You'll need to have an account created for you. If you currently have SVN access, then you have an account, but need to have it linked to Labs. We are still working out the account activation process, but hope to have it done soon.


This page explains how to gain access to [[Portal:Cloud VPS|Cloud VPS]] using SSH.
== Logged-in users ==


== What you'll need ==
After creating an account, you can:
{{tracked|T347637}}


=== Required accounts ===
* [[Special:NovaKey|Add and manage your public SSH keys]]


{{Account_setup}}
Once you add a key, you'll be able to log into the instance of any project you are a member of. Have a current project member, or an admin user add you to a project.


=== Set up and upload SSH keys ===
To create instances within a project, you'll need to have a sysadmin on the project, or an admin add you to the sysadmin role for that project.
# [[Generate an SSH Key]]
# [[Special:Preferences#mw-prefsection-openstack|Upload your public SSH key to Wikitech]]
# [https://gerrit.wikimedia.org Upload your public SSH key Gerrit]


=== Be a member of a Cloud VPS project ===
You can make [[Resource query examples|queries]] for nova resources; currently only Nova instances have semantic properties enabled.


In order to SSH into instances of a particular Cloud VPS project, you must be a member of that project.
After logging in, you can also access [https://gerrit.wikimedia.org Gerrit]; if you wish to do git checkouts of the puppet repositories, you'll need to log into [https://gerrit.wikimedia.org Gerrit], and add your SSH key there as well. Note: it would be nice if Gerrit could use LDAP for its SSH keystore, instead of its database; I've opened a [http://code.google.com/p/gerrit/issues/detail?id=1124 bug for this], if you'd like to help, please add that feature to Gerrit!
In order to SSH even into a bastion, you need to be a member of at least one project (then the <code>project-bastion</code> LDAP group will be added automatically).
[[Help:Cloud VPS project#Request a new Cloud VPS project|Request a new Cloud VPS project]], or ask someone to add you to their existing project.


== Admins ==
== SSH Recommendations ==


=== Wiki Admin ===
=== Linux or macOS ===
* Natively support SSH. You should be able to SSH from the terminal.


=== Windows 10 ===
If you are a wiki admin, you can:
* Windows 10 (Spring 2018 Creators update or higher) has a built in SSH client.
** If the OpenSSH client is not already enabled, you can do this by following <code>Settings</code> '''->''' <code>Apps & features </code> '''->''' <code>Optional features</code> '''->''' <code>Add a feature</code>. Scroll down and enable the SSH Client.
** Access the SSH client via Windows Powershell using the <code>ssh</code> directive.
** To use an SSH agent, you will need to enable it.
*** Type into your search bar <code>services.msc</code> and open the Services program
*** Find OpenSSH Authentication Agent and set that service to "Automatic" and start it if it is disabled.


=== Older versions of Windows ===
* [[Special:NovaProject|Create/delete Nova projects and manage project and project role membership]]
It is recommended that you run the most current version of Windows. However, if you choose to run an older version, you will need an SSH client. [https://www.putty.org/ PuTTY] / [http://kitty.9bis.net/ KiTTY] is often recommended.
* [[Special:NovaRole|Manage global role membership]].
* [[Special:NovaSudoer|Manage sudo policies]].


== Accessing Cloud VPS instances ==
=== Net Admins ===


=== Key concepts ===
If you are a NetAdmin, you can:


; {{anchor|Bastion host}} [[w:Bastion host|Bastion host]]: An instance you use to access other instances. Most instances do not have floating IP addresses assigned, due to our shortage of public IPs. To access them, it's necessary to go through a bastion host as an intermediary. For example <tt>bastion.wmcloud.org</tt> is accessible by every Cloud VPS account holder who has been added to the [[Nova Resource:Bastion|bastion project]]. There are other bastion hosts, e.g. to access Toolforge. See also [[Bastion]].
* [[Special:NovaAddress|Allocate and associate public IP addresses, and add public DNS entries to instances]]
* [[Special:NovaSecurityGroup|Manage security groups and rules]].


; {{anchor|Bastion Instance}} Bastion instance: For security purposes most Cloud VPS instances cannot be directly accessed from the Internet. A bastion instance is used to gain ssh access to other instances. The Cloud VPS bastion (bastion.wmcloud.org) is accessible by every Wikimedia developer account holder who is a member of a Cloud VPS project. Toolforge members are not automatically granted access to the shared Cloud VPS bastion as Toolforge has its own bastion servers (for example: login.toolforge.org).
=== Sys Admins ===


=== Setup ===
If you are a sysadmin, you can:
{{Note|[[Portal:Toolforge|Toolforge]] has [[Portal:Toolforge/About_Toolforge#Bastion_hosts|its own bastions]] and does not require the below configuration.}}
You'll need to proxy through a machine that is visible to the Internet and recognizes Cloud VPS (bastion) instances.


{| class="wikitable"
* [[Special:NovaInstance|Create or manage instances]].
|+How should you proxy?
* [[Special:NovaVolume|Create or manage volumes]].
!Your role
!Use
|-
|A member of Wikimedia SRE Team
|<code>restricted.bastion.wmcloud.org</code>
|-
|Everyone else (including volunteers and Wikimedia Foundation staff)
|<code>primary.bastion.wmcloud.org</code><br/><code>bastion.wmcloud.org</code> (alias)
|}


Configure your <code>$HOME/.ssh/config</code> file to instruct SSH to use <code>bastion.wmcloud.org</code> as a jump host when connecting to <code>*.wikimedia.cloud</code> instances:
After creating an instance, you'll get an email notifying you that it is ready to be logged into. If you did not add an SSH key prior to creating the instance, you'll need to wait until your key is propagated to the instances (which can take an additional 30 minutes).


<syntaxhighlight lang="apache">
=== Cloud Admins ===
Host *.wmflabs.org *.wmcloud.org *.toolforge.org
User <your-shell-name>


Host *.wmflabs *.wikimedia.cloud
In addition to all actions that sysadmins and netadmins, you can:
User <your-shell-name>
ProxyJump bastion.wmcloud.org:22
</syntaxhighlight>


With the above config you can use <code>ssh <your-instance>.<your-project>.eqiad1.wikimedia.cloud</code> to connect to an instance.
* [[Special:NovaDomain|Create or manage DNS domains]].


If you can't or prefer to not alter SSH config files, you can also use the following command to specify the settings all in a longer ssh command:
== Initial log in ==
<syntaxhighlight lang="shell-session">
$ ssh -J <your-shell-name>@bastion.wmcloud.org <your-shell-name>@<your-instance>.<your-project>.eqiad1.wikimedia.cloud
</syntaxhighlight>


=== Logging in ===
If someone has made an account for you, you'll need to do the following for initial log in:
Run the following from your local computer, substituting the instance and project names as appropriate:
ssh ''your-instance''.''your-project''.eqiad1.wikimedia.cloud


==== SSH fingerprints ====
# Go to the [[Special:UserLogin|login page]]
See [[Help:SSH Fingerprints]] for host key fingerprints which can be used to validate the authenticity of keys offered by hosts when attempting to connect for the first time or if the key has changed due to a full reimaging of the server. It is good practice to verify the SSH fingerprint of the bastions you use in order to reduce the likelihood of a [[:en:Man-in-the-middle_attack|MITM attack]].
# Click "Forgotten your login details?"
#* If the "Forgotten your login details?" link does not appear, click "Log in"
# Go through the process of resetting your password
# After logging in, you should [[Special:NovaKey|upload an ssh key to labs]]
# Now log into [https://gerrit.wikimedia.org gerrit]. It'll ask you to upload a key. Upload your public key here as well.


SSH fingerprints of non-bastion servers are usually not listed there; if you can't find a way to get their fingerprint from elsewhere (e.g., it might be printed to the log on first boot, which you can see in Horizon if the instance was newly created), then it's probably fine to accept the host key you connect to it (trust on first use), since the risk of a MITM attack between the instance and the bastion should be lower than between the bastion and your client.
== Accessing public and private instances ==


=== Using agent forwarding ===
== File managers ==
You can connect to your Cloud VPS instance through the bastion via SSH with a file manager. There are a number of Open Source options listed below.


'''Note:''' The following options are maintained by third parties. Please see the technical documentation or readme on the software's website to determine the best method of connection.
Labs has a bastion instance that can be used to access all other instances called ''bastion.wmflabs.org''. To reach nodes from the bastion instance, you'll need to use an SSH agent, and forward that agent when connecting to the bastion instance. For example:


=== Options ===
* On your local system:
eval `ssh-agent`
ssh-add
ssh -A <username>@bastion.wmflabs.org
* On bastion:
ssh <your-instance>.pmtpa.wmflabs


'''Windows'''
The above would let you ssh into your instance. Notice that FQDN isn't completely necessary. You can also just use "ssh <your-instance>".
* [https://www.putty.org/ PuTTY]


'''Linux'''
=== Using ProxyCommand ssh option ===
* Gnome: ([https://wiki.gnome.org/Apps/Files Files, formerly Nautilus]),
* KDE: [https://kde.org/applications/system/org.kde.dolphin Dolphin],
* FUSE: [https://github.com/libfuse/libfuse libfuse on GitHub]


'''Mac'''
Using this configuration, it's possible to directly connect to your instances.
* [https://github.com/libfuse/sshfs SSHFS]


== Troubleshooting ==
Add this to the bottom of your ~/.ssh/config on your local system:
In general, adding SSH option -v, -vv, or -vvv may help identify possible issues.


=== Into Bastion ===
Host bastion1.pmtpa.wmflabs
Hostname bastion.wmflabs.org
ProxyCommand none
Host *.*.wmflabs
ProxyCommand ssh -e none bastion1.pmtpa.wmflabs exec nc -w 3600 %h %p
User <labs-shell-name>
## If you are using a non-default key, you should uncomment the next two lines, and enter
## your key's filename
#IdentityFile <key-filename>
#IdentitiesOnly yes


===== Permission denied (publickey) =====
Now you should be able to run the following directly from your local system (even without agent forwarding):
# Make sure you have uploaded the correct SSH key to [[Special:Preferences#mw-prefsection-openstack|your preferences]]
# Use lowercase letters for your username
# Your SSH user name is your '''instance shell account name''' name (see [[Special:Preferences|User Profile]] > Basic Information in your Wikitech account's Preferences page). It is not necessarily the same as your account's '''username'''


===== Connection closed by remote host =====
ssh <your-instance>.pmtpa.wmflabs
* Make sure you have uploaded the correct SSH key to [[Special:Preferences#mw-prefsection-openstack|your preferences]]
* If you have access to other SSH servers, can you connect to them? If not, then there may be an issue with your SSH client.
* If you use Windows, is Pageant (PuTTY authentication agent) set up with correct keys and running?


===== Blocking connection on OS X with no error message =====
=== Accessing services using port forwarding ===


If you are running OS X and your SSH connection blocks without any error message (while pinging the server works), try
If your instance doesn't have a public IP address, but you need to access a service on it, you can use port forwarding. For instance, here's how you would access a web service:
<code>unset SSH_AUTH_SOCK</code>, and then SSH again. This will unset the socket to ssh-agent.


=== Into ''your-instance'' ===
* On your local system:
ssh <username>@bastion.wmflabs.org -L 8080:<your-instance>:80


===== Permission denied (publickey) =====
You can then connect to it via ''http://localhost:8080''.
* Make sure the instance build has completed.
* Search in the console output for ''“Finished puppet run”'', ''BEGIN SSH HOST KEY FINGERPRINTS'', and ''BEGIN SSH HOST KEY KEYS''.


{{:Help:Cloud Services communication}}
=== Accessing web services using a SOCKS proxy ===


[[Category:Cloud VPS]]
If you only need to access web services on an instance that doesn't have a public IP address, it's easiest to use a SOCKS proxy:

* On your local system:
ssh <username>@bastion.wmflabs.org -D 8080

After doing so, you can configure your browser to use localhost:8080 as a SOCKS proxy, then you can connect to your node directly in your browser, via ''http://<your-instance>.pmtpa.wmflabs''

If you use [http://getfoxyproxy.org/index.html foxy proxy] (a Firefox, IE or Chrome add-on), you can use the pre-defined url pattern defined [https://labsconsole.wikimedia.org/wiki/MediaWiki:Foxy-proxy-labs?action=raw here].

== Puppet ==

Labs instance run from the operations/puppet repository and branch 'test'.

To trigger a puppet update, run: <tt>sudo puppetd -tv</tt>

The cron job that updates the master puppet repo in the labs cluster runs every minute. So your change got merged in, you will have to wait a bit.

== Access FAQ ==

* '''Q:''' I was added to a group that gives me access to something in git, but it isn't working, what's wrong?
** '''A:''' Once you have been added to the group, you need to log out of gerrit, then back in. Gerrit pulls its groups from LDAP, but caches them. Logging out, then back in re-synchronizes your groups, and thus clears the cache.

Revision as of 21:28, 28 September 2023

Overview

This page explains how to gain access to Cloud VPS using SSH.

What you'll need

Tracked in Phabricator
Task T347637

Required accounts

To access Toolforge, Cloud VPS, or other Wikimedia developer tools, you need the following two types of accounts:

Account Type Description Where to sign up
Wikimedia account Also known as "Wikimedia single user login (SUL)" account. Allows you to log into general wikis like Wikipedia, MediaWiki, and MetaWiki. Create Wikimedia account
Wikimedia developer account Username (also called LDAP username) is used to log in on Wikitech (this wiki), Gerrit, and the Toolforge admin console. In GitLab it is your displayed name, so you may want it to be your full name. You will not be able to change this later!

Shell username is used to log into Toolforge, other Wikimedia VPS or Wikimedia production hosts using SSH.

Create Wikimedia developer account

Set up and upload SSH keys

  1. Generate an SSH Key
  2. Upload your public SSH key to Wikitech
  3. Upload your public SSH key Gerrit

Be a member of a Cloud VPS project

In order to SSH into instances of a particular Cloud VPS project, you must be a member of that project. In order to SSH even into a bastion, you need to be a member of at least one project (then the project-bastion LDAP group will be added automatically). Request a new Cloud VPS project, or ask someone to add you to their existing project.

SSH Recommendations

Linux or macOS

  • Natively support SSH. You should be able to SSH from the terminal.

Windows 10

  • Windows 10 (Spring 2018 Creators update or higher) has a built in SSH client.
    • If the OpenSSH client is not already enabled, you can do this by following Settings -> Apps & features -> Optional features -> Add a feature. Scroll down and enable the SSH Client.
    • Access the SSH client via Windows Powershell using the ssh directive.
    • To use an SSH agent, you will need to enable it.
      • Type into your search bar services.msc and open the Services program
      • Find OpenSSH Authentication Agent and set that service to "Automatic" and start it if it is disabled.

Older versions of Windows

It is recommended that you run the most current version of Windows. However, if you choose to run an older version, you will need an SSH client. PuTTY / KiTTY is often recommended.

Accessing Cloud VPS instances

Key concepts

Bastion host
An instance you use to access other instances. Most instances do not have floating IP addresses assigned, due to our shortage of public IPs. To access them, it's necessary to go through a bastion host as an intermediary. For example bastion.wmcloud.org is accessible by every Cloud VPS account holder who has been added to the bastion project. There are other bastion hosts, e.g. to access Toolforge. See also Bastion.
Bastion instance
For security purposes most Cloud VPS instances cannot be directly accessed from the Internet. A bastion instance is used to gain ssh access to other instances. The Cloud VPS bastion (bastion.wmcloud.org) is accessible by every Wikimedia developer account holder who is a member of a Cloud VPS project. Toolforge members are not automatically granted access to the shared Cloud VPS bastion as Toolforge has its own bastion servers (for example: login.toolforge.org).

Setup

Toolforge has its own bastions and does not require the below configuration.

You'll need to proxy through a machine that is visible to the Internet and recognizes Cloud VPS (bastion) instances.

How should you proxy?
Your role Use
A member of Wikimedia SRE Team restricted.bastion.wmcloud.org
Everyone else (including volunteers and Wikimedia Foundation staff) primary.bastion.wmcloud.org
bastion.wmcloud.org (alias)

Configure your $HOME/.ssh/config file to instruct SSH to use bastion.wmcloud.org as a jump host when connecting to *.wikimedia.cloud instances: 

Host *.wmflabs.org *.wmcloud.org *.toolforge.org
  User <your-shell-name>

Host *.wmflabs *.wikimedia.cloud
  User <your-shell-name>
  ProxyJump bastion.wmcloud.org:22

With the above config you can use ssh <your-instance>.<your-project>.eqiad1.wikimedia.cloud to connect to an instance.

If you can't or prefer to not alter SSH config files, you can also use the following command to specify the settings all in a longer ssh command: 

$ ssh -J <your-shell-name>@bastion.wmcloud.org <your-shell-name>@<your-instance>.<your-project>.eqiad1.wikimedia.cloud

Logging in

Run the following from your local computer, substituting the instance and project names as appropriate: 

ssh your-instance.your-project.eqiad1.wikimedia.cloud

SSH fingerprints

See Help:SSH Fingerprints for host key fingerprints which can be used to validate the authenticity of keys offered by hosts when attempting to connect for the first time or if the key has changed due to a full reimaging of the server. It is good practice to verify the SSH fingerprint of the bastions you use in order to reduce the likelihood of a MITM attack.

SSH fingerprints of non-bastion servers are usually not listed there; if you can't find a way to get their fingerprint from elsewhere (e.g., it might be printed to the log on first boot, which you can see in Horizon if the instance was newly created), then it's probably fine to accept the host key you connect to it (trust on first use), since the risk of a MITM attack between the instance and the bastion should be lower than between the bastion and your client.

File managers

You can connect to your Cloud VPS instance through the bastion via SSH with a file manager. There are a number of Open Source options listed below.

Note: The following options are maintained by third parties. Please see the technical documentation or readme on the software's website to determine the best method of connection.

Options

Windows

Linux

Mac

Troubleshooting

In general, adding SSH option -v, -vv, or -vvv may help identify possible issues.

Into Bastion

Permission denied (publickey)
  1. Make sure you have uploaded the correct SSH key to your preferences
  2. Use lowercase letters for your username
  3. Your SSH user name is your instance shell account name name (see User Profile > Basic Information in your Wikitech account's Preferences page). It is not necessarily the same as your account's username
Connection closed by remote host
  • Make sure you have uploaded the correct SSH key to your preferences
  • If you have access to other SSH servers, can you connect to them? If not, then there may be an issue with your SSH client.
  • If you use Windows, is Pageant (PuTTY authentication agent) set up with correct keys and running?
Blocking connection on OS X with no error message

If you are running OS X and your SSH connection blocks without any error message (while pinging the server works), try unset SSH_AUTH_SOCK, and then SSH again. This will unset the socket to ssh-agent.

Into your-instance

Permission denied (publickey)
  • Make sure the instance build has completed.
  • Search in the console output for “Finished puppet run”, BEGIN SSH HOST KEY FINGERPRINTS, and BEGIN SSH HOST KEY KEYS.

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support
Stay aware of critical changes and plans
Track work tasks and report bugs

Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself

Read stories and WMCS blog posts

Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Help:Accessing_Cloud_VPS_instances&oldid=2116067"