Mix nodes
The Nym mix node binary was built in the building nym section. If you haven't yet built Nym and want to run the code, go there first.
After your build is finished, the nym-mixnode binary will be located in /path/to/nym/target/release/ directory. You may move or copy it to wherever you wish (for example, you may wish to compile your binaries once locally and then move them to different machines).
Alternatively, you can fetch the binaries from our releases page.
Preliminary steps
There are a couple of steps that need completing before starting to set up your mix node:
- preparing your wallet
- requisitioning a VPS (Virtual Private Server)
Wallet preparation
Mainnet
Before you initialise and run your mixnode, head to our website and download the Nym wallet for your operating system. If pre-compiled binaries for your operating system aren't availiable, you can build the wallet yourself with instructions here.
If you don't already have one, please create a Nym address using the wallet, and fund it with tokens. The minimum amount required to bond a mixnode is 100 NYM, but make sure you have a bit more to account for gas costs.
NYM is currently present on several exchanges. Head to our telegram channels to find out where to get NYM tokens.
Remember that you can only use Cosmos NYM tokens to bond your mixnode. You cannot use ERC20 representations of NYM to run a node.
Sandbox testnet
Make sure to download a wallet and create an account as outlined above. Then head to our token faucet and get some tokens to use to bond it.
VPS Hardware Specs
You will need to rent a VPS to run your mix node on. One key reason for this is that your node must be able to send TCP data using both IPv4 and IPv6 (as other nodes you talk to may use either protocol.
For the moment, we haven't put a great amount of effort into optimizing concurrency to increase throughput, so don't bother provisioning a beastly server with multiple cores. This will change when we get a chance to start doing performance optimizations in a more serious way. Sphinx packet decryption is CPU-bound, so once we optimise, more fast cores will be better.
For now, see the below rough specs:
- Processors: 2 cores are fine. Get the fastest CPUs you can afford.
- RAM: Memory requirements are very low - typically a mix node may use only a few hundred MB of RAM.
- Disks: The mixnodes require no disk space beyond a few bytes for the configuration files.
Mix node setup and maintenance
Now that you have built the codebase, set up your wallet, and have a VPS with the nym-mixnode binary, you can set up your mix node with the instructions below.
Viewing command help
You can check that your binaries are properly compiled with:
./nym-mixnode --help
Which should return a list of all avaliable commands.
console output
_ __ _ _ _ __ ___
| '_ \| | | | '_ \ _ \
| | | | |_| | | | | | |
|_| |_|\__, |_| |_| |_|
|___/
(mixnode - version 1.1.0)
nym-mixnode 1.1.0
Nymtech
Implementation of a Loopix-based Mixnode
USAGE:
nym-mixnode [OPTIONS] <SUBCOMMAND>
OPTIONS:
--config-env-file <CONFIG_ENV_FILE>
Path pointing to an env file that configures the mixnode
-h, --help
Print help information
-V, --version
Print version information
SUBCOMMANDS:
completions Generate shell completions
describe Describe your mixnode and tell people why they should delegate state to
you
generate-fig-spec Generate Fig specification
help Print this message or the help of the given subcommand(s)
init Initialise the mixnode
node-details Show details of this mixnode
run Starts the mixnode
sign Sign text to prove ownership of this mixnode
upgrade Try to upgrade the mixnode
You can also check the various arguments required for individual commands with:
./nym-mixnode <command> --help
Initialising your mix node
To check available configuration options for initializing your node use:
./nym-mixnode init --help
console output
_ __ _ _ _ __ ___
| '_ \| | | | '_ \ _ \
| | | | |_| | | | | | |
|_| |_|\__, |_| |_| |_|
|___/
(mixnode - version 1.1.0)
nym-mixnode-init
Initialise the mixnode
USAGE:
nym-mixnode init [OPTIONS] --id <ID> --host <HOST> --wallet-address <WALLET_ADDRESS>
OPTIONS:
--announce-host <ANNOUNCE_HOST>
The custom host that will be reported to the directory server
-h, --help
Print help information
--host <HOST>
The host on which the mixnode will be running
--http-api-port <HTTP_API_PORT>
The port on which the mixnode will be listening for http requests
--id <ID>
Id of the mixnode we want to create config for
--mix-port <MIX_PORT>
The port on which the mixnode will be listening for mix packets
--validators <VALIDATORS>
Comma separated list of rest endpoints of the validators
--verloc-port <VERLOC_PORT>
The port on which the mixnode will be listening for verloc packets
--wallet-address <WALLET_ADDRESS>
The wallet address you will use to bond this mixnode, e.g.
nymt1z9egw0knv47nmur0p8vk4rcx59h9gg4zuxrrr9
Initalise your mixnode with the following command, replacing the value of --id with the moniker you wish to give your mixnode, and the --wallet-address with the Nym address you created earlier:
./nym-mixnode init --id winston-smithnode --host $(curl ifconfig.me) --wallet-address <wallet-address>
Your --host must be publicly routable on the internet in order to mix packets, and can be either an Ipv4 or IPv6 address.
Your node must be able to send TCP data using both IPv4 and IPv6 (as other nodes you talk to may use either protocol).
The $(curl ifconfig.me) command above returns your IP automatically using an external service. Alternatively, you can enter your IP manually wish. If you do this, remember to enter your IP without any port information.
The init command will refuse to destroy existing mix node keys.
During the init process you will have the option to change the http_api, verloc and mixnode ports from their default settings. If you wish to change these in the future you can edit their values in the config.toml file created by the initialization process, which is located at ~/.nym/mixnodes/<your-id>/.
Bonding your mix node
From this release, if you unbond your mixnode that means you are leaving the mixnet and you will lose all your delegations (permanently). You can join again with the same identity key, however, you will start with no delegations.
Via the Desktop wallet (recommended)
You can bond your mix node via the Desktop wallet.
Open your wallet, and head to the Bond page, then select the node type and input your node details.
Via the CLI (power users)
If you want to bond your Gateway via the CLI, then check out the Nym CLI tool.
Running your mix node
Now you've bonded your mix node, run it with:
./nym-mixnode run --id winston-smithnode
console output
Starting mixnode winston-smithnode...
To bond your mixnode you will need to install the Nym wallet, go to https://nymtech.net/get-involved and select the Download button.
Select the correct version and install it to your machine. You will need to provide the following:
Identity Key: GWrymUuLaxVHSs8iE7YW48MB81npnKjrVuJzJsGkeji6
Sphinx Key: FU89ULkS4YYDXcm5jShhJvoit7H4jG4EXHxRKbS9cXSJ
Owner Signature: Kd5StZtg5PsjLtWRJ5eQejuLHz3JUNzZrk6Jd4WVS5u9Q5bFt6DvuVzN7NbiX9WMZYpsYMJoegH3Bz94o6gsY6b
Host: 62.240.134.46 (bind address: 62.240.134.46)
Version: 1.1.0
Mix Port: 1789, Verloc port: 1790, Http Port: 8000
You are bonding to wallet address: n1x42mm3gsdg808qu2n3ah4l4r9y7vfdvwkw8az6
2022-04-27T16:08:01.159Z INFO nym_mixnode::node > Starting nym mixnode
2022-04-27T16:08:01.490Z INFO nym_mixnode::node > Starting node stats controller...
2022-04-27T16:08:01.490Z INFO nym_mixnode::node > Starting packet delay-forwarder...
2022-04-27T16:08:01.490Z INFO nym_mixnode::node > Starting socket listener...
2022-04-27T16:08:01.490Z INFO nym_mixnode::node::listener > Running mix listener on "62.240.134.46:1789"
2022-04-27T16:08:01.490Z INFO nym_mixnode::node > Starting the round-trip-time measurer...
If everything worked, you'll see your node running on the either the Sandbox testnet network explorer or the mainnet network explorer, depending on which network you're running.
Note that your node's public identity key is displayed during startup, you can use it to identify your node in the list.
Keep reading to find out more about configuration options or troubleshooting if you're having issues. There are also some tips for running on AWS and other cloud providers, some of which require minor additional setup.
Also have a look at the saved configuration files in $HOME/.nym/mixnodes/ to see more configuration options.
Describe your mix node (optional)
In order to easily identify your node via human-readable information later on in the development of the testnet when delegated staking is implemented, you can describe your mixnode with the following command:
./nym-mixnode describe --id winston-smithnode
console output
name: winston-smithnode
description: nym-mixnode hosted on Linode VPS in <location> with the following specs: <specs>.
link, e.g. https://mixnode.yourdomain.com: mixnode.mydomain.net
location, e.g. City: London, Country: UK: <your_location>
This information will be shown in the mixnode's page in the Network Explorer, and help people make delegated staking decisions.
Remember to restart your mix node process in order for the new description to be propogated
Upgrading your mix node
- pause your mix node process
- replace the existing binary with the newest binary (which you can either compile yourself or grab from our releases page)
- re-run
initwith the same values as you used initially. This will just update the config file, it will not overwrite existing keys. - restart your mix node process with the new binary.
Do not use the
upgradecommand: there is a known error with the command that will be fixed in the next release.
Displaying mix node information
You can always check the details of your mix node with the node-details command:
./nym-mixnode node-details --id winston-smithnode
console output
Identity Key: GWrymUuLaxVHSs8iE7YW48MB81npnKjrVuJzJsGkeji6
Sphinx Key: FU89ULkS4YYDXcm5jShhJvoit7H4jG4EXHxRKbS9cXSJ
Owner Signature: Kd5StZtg5PsjLtWRJ5eQejuLHz3JUNzZrk6Jd4WVS5u9Q5bFt6DvuVzN7NbiX9WMZYpsYMJoegH3Bz94o6gsY6b
Host: 62.240.134.46 (bind address: 62.240.134.46)
Version: 1.1.0
Mix Port: 1789, Verloc port: 1790, Http Port: 8000
You are bonding to wallet address: n1x42mm3gsdg808qu2n3ah4l4r9y7vfdvwkw8az6
Configure your firewall
The following commands will allow you to set up a firewall using ufw.
# check if you have ufw installed
ufw version
# if it is not installed, install with
sudo apt install ufw -y
# enable ufw
sudo ufw enable
# check the status of the firewall
sudo ufw status
Finally open your mix node's p2p port, as well as ports for ssh, http, and https connections, and ports 8000 and 1790 for verloc and measurement pings:
sudo ufw allow 1789,1790,8000,22,80,443/tcp
# check the status of the firewall
sudo ufw status
For more information about your mix node's port configuration, check the mix node port reference table below.
Automating your mix node with systemd
It's useful to have the mix node automatically start at system boot time. Here's a systemd service file to do that:
[Unit]
Description=Nym Mixnode (1.1.0)
StartLimitInterval=350
StartLimitBurst=10
[Service]
User=nym
LimitNOFILE=65536
ExecStart=/home/nym/nym-mixnode run --id mix0100
KillSignal=SIGINT
Restart=on-failure
RestartSec=30
[Install]
WantedBy=multi-user.target
Put the above file onto your system at /etc/systemd/system/nym-mixnode.service.
Change the path in ExecStart to point at your mix node binary (nym-mixnode), and the User so it is the user you are running as.
If you have built nym on your server, and your username is jetpanther, then the start command might look like this:
ExecStart=/home/jetpanther/nym/target/release/nym-mixnode run --id your-id. Basically, you want the full /path/to/nym-mixnode run --id whatever-your-node-id-is
Then run:
systemctl enable nym-mixnode.service
Start your node:
service nym-mixnode start
This will cause your node to start at system boot time. If you restart your machine, the node will come back up automatically.
You can also do service nym-mixnode stop or service nym-mixnode restart.
Note: if you make any changes to your systemd script after you've enabled it, you will need to run:
systemctl daemon-reload
This lets your operating system know it's ok to reload the service configuration.
Setting the ulimit
Linux machines limit how many open files a user is allowed to have. This is called a ulimit.
ulimit is 1024 by default on most systems. It needs to be set higher, because mix nodes make and receive a lot of connections to other nodes.
If you see errors such as:
Failed to accept incoming connection - Os { code: 24, kind: Other, message: "Too many open files" }
This means that the operating system is preventing network connections from being made.
Set the ulimit via systemd service file
Query the ulimit of your mix node with:
grep -i "open files" /proc/$(ps -A -o pid,cmd|grep nym-mixnode | grep -v grep |head -n 1 | awk '{print $1}')/limits
You'll get back the hard and soft limits, which looks something like this:
Max open files 65536 65536 files
If your output is the same as above, your node will not encounter any ulimit related issues.
However if either value is 1024, you must raise the limit via the systemd service file. Add the line:
LimitNOFILE=65536
Reload the daemon:
systemctl daemon-reload
or execute this as root for system-wide setting of ulimit:
echo "DefaultLimitNOFILE=65535" >> /etc/systemd/system.conf
Reboot your machine and restart your node. When it comes back, use cat /proc/$(pidof nym-mixnode)/limits | grep "Max open files" to make sure the limit has changed to 65535.
Set the ulimit on non-systemd based distributions
Edit etc/security/conf and add the following lines:
# Example hard limit for max opened files
username hard nofile 4096
# Example soft limit for max opened files
username soft nofile 4096
Then reboot your server and restart your mixnode.
Checking that your node is mixing correctly
Network explorers
Once you've started your mix node and it connects to the testnet validator, your node will automatically show up in the 'Mix nodes' section of either the Nym Network Explorers:
Enter your identity key to find your node. There are numerous statistics about your node on that page that are useful for checking your uptime history, packets mixed, and any delegations your node may have.
There are also 2 community explorers which have been created by Nodes Guru:
For more details see Troubleshooting FAQ
Virtual IPs and hosting via Google & AWS
On some services (AWS, Google, etc), the machine's available bind address is not the same as the public IP address. In this case, bind --host to the local machine address returned by ifconfig, but also specify --announce-host with the public IP. Please make sure that you pass the correct, routable --announce-host.
For example, on a Google machine, you may see the following output from the ifconfig command:
ens4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1460
inet 10.126.5.7 netmask 255.255.255.255 broadcast 0.0.0.0
...
The ens4 interface has the IP 10.126.5.7. But this isn't the public IP of the machine, it's the IP of the machine on Google's internal network. Google uses virtual routing, so the public IP of this machine is something else, maybe 36.68.243.18.
nym-mixnode init --host 10.126.5.7, initalises the mix node, but no packets will be routed because 10.126.5.7 is not on the public internet.
Trying nym-mixnode init --host 36.68.243.18, you'll get back a startup error saying AddrNotAvailable. This is because the mix node doesn't know how to bind to a host that's not in the output of ifconfig.
The right thing to do in this situation is nym-mixnode init --host 10.126.5.7 --announce-host 36.68.243.18.
This will bind the mix node to the available host 10.126.5.7, but announce the mix node's public IP to the directory server as 36.68.243.18. It's up to you as a node operator to ensure that your public and private IPs match up properly.
Metrics / API endpoints
The mix node binary exposes several API endpoints that can be pinged in order to gather information about the node, and the Validator API exposes numerous mix node related endpoints which provide network-wide information about mix nodes, the network topology (the list of avaliable mix nodes for packet routing), and information regarding uptime monitoring and rewarding history.
Mix node API endpoints
Since the mix node binary exposes several API endpoints itself, you can ping these easily via curl:
| Endpoint | Description | Command |
|---|---|---|
/description | Returns the description of the node set with the describe command | curl <NODE_IP_ADDRESS>:8000/description |
/hardware | Returns the hardware information of the node | curl <NODE_IP_ADDRESS>:8000/hardware |
/verloc | Returns the verloc information of the node, updated every 12 hours | curl <NODE_IP_ADDRESS>:8000/verloc |
The code for exposed API endpoints can be found here.
Mix node related Validator API endpoints
Numerous endpoints are documented on the Validator API's Swagger Documentation. There you can also try out various requests from your broswer, and download the response from the API. Swagger will also show you what commands it is running, so that you can run these from an app or from your CLI if you prefer.
Ports
All mix node-specific port configuration can be found in $HOME/.nym/mixnodes/<your-id>/config/config.toml. If you do edit any port configs, remember to restart your mix node.
Mix node port reference
| Default port | Use |
|---|---|
1789 | Listen for mixnet traffic |
1790 | Listen for VerLoc traffic |
8000 | Metrics http API endpoint |