Operators
Changelog

Changelog

This page displays a full list of all the changes during our release cycle from v2024.3-eclipse onward. Operators can find here the newest updates together with links to relevant documentation. The list is sorted so that the newest changes appear first.

ℹ️

Our documentation often refer to syntax annotated in <> brackets. We use this expression for variables that are unique to each user (like path, local moniker, versions etcetra). Any syntax in <> brackets needs to be substituted with your correct name or version, without the <> brackets. If you are unsure, please check our table of essential parameters and variables (opens in a new tab).

v2024.14-crunch-patched

Patch for v2024.14-crunch release. Fixes an issue (opens in a new tab) to allow only one private IP pair & compatibility issues between nym-nodes and older clients.

nym-node
Binary Name:        nym-node
Build Timestamp:    2024-12-18T10:18:42.978852430Z
Build Version:      1.2.1
Commit SHA:         8d5a41a790e96ae5e821964865affaa7d3343eab
Commit Date:        2024-12-18T11:07:49.000000000+01:00
Commit Branch:      HEAD
rustc Version:      1.83.0
rustc Channel:      stable
cargo Profile:      release

v2024.14-crunch

nym-node
Binary Name:        nym-node
Build Timestamp:    2024-12-11T13:49:11.974104790Z
Build Version:      1.2.0
Commit SHA:         a491e6a71a8cf862d77defd740a4ee8d65d8292a
Commit Date:        2024-12-11T10:28:47.000000000+01:00
Commit Branch:      HEAD
rustc Version:      1.83.0
rustc Channel:      stable
cargo Profile:      release

Features

PackageFromTo
anyhow (opens in a new tab)1.0.891.0.90
clap (opens in a new tab)4.5.184.5.20
clap_complete (opens in a new tab)4.5.294.5.33
pin-project (opens in a new tab)1.1.51.1.6
serde (opens in a new tab)1.0.2101.0.211
serde_json (opens in a new tab)1.0.1281.0.132
wasm-bindgen (opens in a new tab)0.2.930.2.95
wasm-bindgen-futures (opens in a new tab)0.4.430.4.45
web-sys (opens in a new tab)0.3.700.3.72
Updates anyhow1.0.891.0.90

Bugfix

Operators Updates & Tools

⚠️

Nym Network will now only allow nodes which migrated their node in Nym mixnet smart contract to Nym Node. All nodes which are still bonded as a legacy one (Mixnode or Gateway) in the wallet will have no chance to take part in the Rewarded set selection.

Operators taking part in Delegation program or Service Grant program must migrate their nodes latest by December 16th, 08:00 UTC.

Updates

  • Version count as a part of config score has been introduced. To familiarize yourself with Nym Node operator rewards calculation, read this page.
  • Nym nodes running as Exit Gateway in Service Grant program received delegation. Nym team is now delegating total of 64,800,000 NYM on top 241 Nym Nodes (137 in Mixnode mode and 104 as Gateways). Our delegation aims to incentivise committed operators who support bootstrapping of Nym network before paying users come.

  • 250k NYM - Upgrading to magura in time - 2 nodes
  • 300k NYM - Upgrading to magura + bonus for a quick patch upgrade - 102 nodes
  • No delegation - not upgrading in time - 2 nodes

These commands can be run one by one or copy-pasted and run as a block.

mkdir $HOME/nym-binaries; \
 
curl -L https://raw.githubusercontent.com/nymtech/nym/refs/heads/develop/scripts/network_tunnel_manager.sh -o $HOME/nym-binaries/network_tunnel_manager.sh && \
chmod +x $HOME/nym-binaries/network_tunnel_manager.sh; \
 
$HOME/nym-binaries/network_tunnel_manager.sh check_nymtun_iptables  ; \
$HOME/nym-binaries/network_tunnel_manager.sh remove_duplicate_rules nymtun0 ;\
$HOME/nym-binaries/network_tunnel_manager.sh remove_duplicate_rules nymwg;\
$HOME/nym-binaries/network_tunnel_manager.sh check_nymtun_iptables ; \
$HOME/nym-binaries/network_tunnel_manager.sh adjust_ip_forwarding ; \
$HOME/nym-binaries/network_tunnel_manager.sh apply_iptables_rules ; \
$HOME/nym-binaries/network_tunnel_manager.sh check_nymtun_iptables ; \
$HOME/nym-binaries/network_tunnel_manager.sh apply_iptables_rules_wg ; \
$HOME/nym-binaries/network_tunnel_manager.sh configure_dns_and_icmp_wg ; \
$HOME/nym-binaries/network_tunnel_manager.sh adjust_ip_forwarding ; \
$HOME/nym-binaries/network_tunnel_manager.sh check_ipv6_ipv4_forwarding; \
 
systemctl daemon-reload && service nym-node restart && journalctl -u nym-node -f

Then run the jokes in a new window for control

$HOME/nym-binaries/network_tunnel_manager.sh joke_through_the_mixnet
$HOME/nym-binaries/network_tunnel_manager.sh joke_through_wg_tunnel

Tools

magura-drift

Second patch to v2024.13-magura release version.

nym-node
Binary Name:        nym-node
Build Timestamp:    2024-11-29T13:10:51.813092288Z
Build Version:      1.1.12
Commit SHA:         4a9a5579c40ad956163ea02e01d7b53aef2ac8ef
Commit Date:        2024-11-29T14:06:32.000000000+01:00
Commit Branch:      HEAD
rustc Version:      1.83.0
rustc Channel:      stable
cargo Profile:      release
  • This patch adds a peer storage manager to fix issues causing external clients to be blocked, ensuring they can successfully connect to different nodes.

v2024.13-magura-patched

nym-node
Binary Name:        nym-node
Build Timestamp:    2024-11-22T14:30:48.067329245Z
Build Version:      1.1.11
Commit SHA:         01c7b2819ee3d328deccd303b4113ff415d7e276
Commit Date:        2024-11-22T10:50:59.000000000+01:00
Commit Branch:      HEAD
rustc Version:      1.82.0
rustc Channel:      stable
cargo Profile:      release
⚠️

After changes coming along with v2024.13-magura (nym-node v1.1.10), Nym Explorer is no longer picking all values correctly. Instead of fixing this outdated explorer, we are working on a new one, coming out soon.

Nym Harbourmaster (opens in a new tab) has cache of 90min, expect your values to be updated with delay. We are aware of some issues with Nym Harbourmaster and working hard to resolve them in the upcoming explorer v2. To check your routing values in real time, you can use nym-gateway-probe.

Operators Updates & Tools

1. Open your node config file:
nano $HOME/.nym/nym-nodes/<ID>/config/config.toml
 
# change <ID> for your local nym moniker for example:
# nano $HOME/.nym/nym-nodes/default-nym-node/config/config.toml
2. Control or change the value of wireguard private IP
  • Scroll down to section starting with [wireguard]
  • Find line private_ip and ensure it's set to value 10.1.0.1
  • The section will look like this:
[wireguard]
# Specifies whether the wireguard service is enabled on this node.
enabled = true
 
# Socket address this node will use for binding its wireguard interface.
# default: `0.0.0.0:51822`
bind_address = '0.0.0.0:51822'
 
# Private IP address of the wireguard gateway.
# default: `10.1.0.1`
private_ip = '10.1.0.1'
3. Save, exit and restart the service
  • If you used nano editor - press ctrl + x and confirm the changes
  • Run these commands to update the service with new values and restart your node process:
systemctl daemon-reload && service nym-node restart

  • New manual how to run nym-node as non-root

  • Since v2024.13-magura, operators do not update their node version in the wallet. Manual upgrading steps has been updated accordingly.

  • CLI tool node_api_check.py, helping operators to collect all API values about their nodes locally, is not up to date with the API changes introduced with v2024.13-magura release version. Please treat it as unstable before we fix it.

Error Log

In case you encounter this error:

[ERROR] nym-node/src/node/mod.rs:628: the exit gateway subtask has failed with the following message: failed to start authenticator: internal wireguard error no private IP set for peer..

You can follow these steps to make a workaround:


1. Find the error

  • In the node logs, locate the ERROR message which says the exit gateway subtask has failed with the following message: failed to start authenticator: internal wireguard error no private IP set for peer KN5GPvkC+p6G/SM4PD2Z3ObAtRGiDjHPRnQOPpbdUQk=

  • Copy the end part of that peer, later denoted as <WG_PEER_STRING_END> (in our example GiDjHPRnQOPpbdUQk=) to use later in the sql commands

2. Fix the issue in sqlite3 db
⚠️

Be careful when running commands within sqlite database.

  • Navigate to the data directory:
cd $HOME/.nym/nym-nodes/<ID>/data
  • Enter the database:
sqlite3 clients.sqlite
  • Run these commands:
# Change with your unique <PEER_STRING_END>
select * from wireguard_peer where public_key like "%<WG_PEER_STRING_END>%"
# Make sure that only ONE line is returned and it contains the key
 
delete from wireguard_peer where public_key like "%<WG_PEER_STRING_END>%";
  • Confirm that peer has been removed by running this again:
select * from wireguard_peer where public_key like "%<WG_PEER_STRING_END>%";
3. Exit and restart the service

Run .quit and:

systemctl restart nym-node.service

v2024.13-magura

Magura release represents a bigger milestone in project Smoosh development where nym-node is one binary able to perform any function in Nym Mixnet. This release is especially crucial for operators, please pay attention to the section Operators Updates & Tooling below.

nym-node
Binary Name:        nym-node
Build Timestamp:    2024-11-18T17:02:50.947941194Z
Build Version:      1.1.10
Commit SHA:         b49ef643df86f0c670672429812c632fbbaf6cf1
Commit Date:        2024-11-18T17:56:57.000000000+01:00
Commit Branch:      HEAD
rustc Version:      1.82.0
rustc Channel:      stable
cargo Profile:      release

Features

Confirm that the deployment workflows work through manual testing

  • cd-docs
  • publish-sdk-npm

Bugfix

Operators Updates & Tooling

⚠️

Every operator has to make sure that their nodes self-described endpoint works, otherwise the node will be un-routable and thus won't get any rewards!

Wallet changes

New wallet version 1.2.15 (opens in a new tab) is out!


1. Download the wallet from the release page (opens in a new tab)
2. Verify the binary and extract it if needed
  • Download hashes.json (opens in a new tab)
  • Open it with your text editor or print it's content with cat hashes.json
  • Run sha256sum <WALLET_BINARY> for example sha256sum ./nym-wallet_1.2.15_amd64.AppImage
  • If your have to extract it (like .tar.gz) do it
3. Open the wallet and sign in
4. Migrate!
  • Go to Bonding and you will be prompted with such message:

  • In case you for some reason didn't see the prompt or you closed it - you can click in the upper right corner of the same window on this button:

  • Confirm the transaction
5. Welcome to new episode of nym-node!
  • Older versions will not allow bonding new nodes!

Selection & Rewarding

  • Config score is introduced: In the current version the nodes selection to the active set has a new parameter (which multiplies the existing formula) - config_score. Config score looks if the node binary is nym-node (not legacy nym-mixnode or nym-gateway) AND if Terms & Conditions are accepted. Config score has binary values of either 0 or 1, with a following logic:
Run nym-node binaryT&C's acceptedconfig_score
TrueFalse0
FalseTrue0
FalseFalse0
TrueTrue1
  • The active set selection formula is then:

CONFIG_SCORE * STAKE_SATURATION * PERFORMANCE ^ 20

  • Currently in Native rewarding, the rewards are split equally across the rewarded set of nodes (opens in a new tab) (which now = active set and it's size is 240 nodes) for both Mixnet mode and dVPN mode. Every node being assigned 1 / 240 work factor (hence naive rewarding).

Directory Services v2.1: API & Mixnet Contract Changes

Magura release brings breaking changes on API (opens in a new tab) logic of Nym. New APIs will only communicate with nym-node from this release and newer. Also old version of APIs won't be able to communicate with the new version of nym-node. We are also moving towards completely removing Nym Explorer API, which now has been only used to report nodes location.

Any new bonded node will provide only the bare minimum information: host, identity key and optionally custom port of its HTTP api - we highly recommend to set that one up to 8080. Everything else will be discovered via the self-described API for maximum flexibility. This also includes the sphinx key, meaning if the API is not exposed, the node will be unable to route any traffic. Furthermore, this allows to arbitrary change of nym-node from mixnode into a gateway modes (and vice versa) without losing any delegations.

The contract changes also mean any node functionality can get rewards. Rather than just with assigned mixing roles, gateways now also added into the pool. However, to be eligible for gateway rewarding, one must migrate into a nym-node on a smart contract level (or bond a new node).

API High Level Changes
New/Added
  • All new routes that return multiple nodes/entries/etc now wrap their responses to expect pagination. Currently, however, full data is returned for each of the endpoints since the pagination hasn't been implemented yet. But once we add it, it won't be a breaking API change.
Removed
  • rocket support has been completely removed. All routes are now always served via axum
Changed
  • Getting anything to do with all nodes (including gateways) requires knowing their node_id. For legacy gateway endpoints, we have a helper method that translates identity key to the node_id
  • Rewarded set is no longer populated with just mixnodes. Instead nym-nodes are assigned to eligible roles (based on stake and performance) in the following order:
    • entry gateways
    • exit gateways
    • mixnodes
    • standby
  • A lot of legacy routes got deprecated. while technically they still "work" and return data, they only return data for legacy nym-mixnode and nym-gateway. What it means is that as operators are migrating their nodes (in the smart contract), those endpoints will start running dry.
  • Since layers are only assigned during rewarded set assignment, for the purposes of network monitor (v1) and legacy mixnode routes, layerless nodes are put on random layers during annotation
  • All legacy gateway queries now also include additional field in their respones: node_id that indicate the id pre-assigned during contract migration
  • Nym Node performance is a bit odd. When network monitors (v1 and v2) were made, there was no concept of a Nym Node. The solution taken is checking whther there is any mixnode performance for node with a given id, if so - return it. Otherwise we grab the equivalent gateway performance. In the future it should probably be averaged or maybe split into explicit mixing or gateway routing performance metrics.
nym-api Changes
  • Root route / now redirects to /swagger
nym-node Routes

  • /v1/nym-nodes/annotation/<NODE_ID> - get annotation about particular nym-node, as gathered by this nym-api. Currently this just includes last 24h performance metric and the current node role
  • /v1/nym-nodes/bonded - get bond information about Nym Nodes, as present in the mixnet contract
  • /v1/nym-nodes/described - get described information about Nym Nodes, as present on their self-described API
  • /v1/nym-nodes/historical-performance/<NODE_ID> - return historical performance of this nym-node on the provided date
  • /v1/nym-nodes/performance-history/<NODE_ID> - return performance history of this nym-node (as a 0 - 1 float)
  • /v1/nym-nodes/uptime-history/<NODE_ID> - return current uptime of this nym-node (as a 0 - 100 u8); added for compatibility with existing APIs using that data format
  • /v1/nym-nodes/performance/<NODE_ID> - return current performance of this nym-node

  • /v1/unstable/nym-nodes/noise - returns basic information needed for the noise protocol between nodes
  • /v1/unstable/nym-nodes/skimmed/active - returns all: Nym Nodes and legacy mixnodes and legacy gateways, that are currently in the active set, unless no-legacy parameter is used
  • /v1/unstable/nym-nodes/skimmed/mixnodes/active - returns all: Nym Nodes and legacy mixnodes, that are currently in the active set, unless no-legacy parameter is used
  • /v1/unstable/nym-nodes/skimmed/mixnodes/all - returns all: Nym Nodes and legacy mixnodes, that are currently bonded and support mixing role, unless no-legacy parameter is used
  • /v1/unstable/nym-nodes/skimmed/entry-gateways/active - returns all: Nym Nodes and legacy gateways, that are currently in the active set and are assigned the entry role, unless no-legacy parameter is used
  • /v1/unstable/nym-nodes/skimmed/exit-gateways/active - returns all: Nym Nodes and legacy gateways, that are currently in the active set and are assigned the exit role, unless no-legacy parameter is used
  • /v1/unstable/nym-nodes/skimmed/entry-gateways/all - returns all: Nym Nodes and legacy gateways, that are currently bonded and support entry gateway role, unless no-legacy parameter is used
  • /v1/unstable/nym-nodes/skimmed/exit-gateways/all - returns all: Nym Nodes and legacy gateways, that are currently bonded and support exit gateway role, unless no-legacy parameter is used
Deprecated (will be removed eventually, so please migrate away from their usage)

Some endpoints got purposely deprecated without any equivalent reimplemented since they do not belong on nym-api. This includes for example /stake-saturation (which can be obtained directly from the contract instead) or /inclusion-probability (for this run your own Monte Carlo simulation).


  • contract-cache routes - all of the below got deprecated as they will only return legacy nym-mixnode and nym-gateway data:

    • /v1/gateways
    • /v1/gateways/blacklisted
    • /v1/mixnodes
    • /v1/mixnodes/active - just to restate the obvious, it will only return a small SUBSET of the active set that since it will ignore active Nym Nodes
    • /v1/mixnodes/active/detailed
    • /v1/mixnodes/blacklisted
    • /v1/mixnodes/detailed
    • /v1/mixnodes/rewarded
    • /v1/mixnodes/rewarded/detailed

  • status routes - all of the below got deprecated as they will only return legacy nym-mixnode and nym-gateway data:

    • /v1/status/gateway/<ID_KEY>/report
    • /v1/status/gateway/<ID_KEY>/history
    • /v1/status/gateway/<ID_KEY>/core-status-count
    • /v1/status/gateway/<ID_KEY>/avg_uptime
    • /v1/status/gateways/detailed
    • /v1/status/gateways/detailed-unfiltered
    • /v1/status/mixnode/<MIX_ID>/report
    • /v1/status/mixnode/<MIX_ID>/history
    • /v1/status/mixnode/<MIX_ID>/core-status-count
    • /v1/status/mixnode/<MIX_ID>/avg_uptime
    • /v1/status/mixnodes/detailed
    • /v1/status/mixnodes/detailed-unfiltered
    • /v1/status/mixnode/<MIX_ID>/status
    • /v1/status/mixnode/<MIX_ID>/reward-estimation
    • /v1/status/mixnode/<MIX_ID>/compute-reward-estimation
    • /v1/status/mixnode/<MIX_ID>/stake-saturation
    • /v1/status/mixnode/<MIX_ID>/inclusion-probability
    • /v1/status/mixnodes/inclusion_probability
    • /v1/status/mixnodes/rewarded/detailed
    • /v1/status/mixnodes/active/detailed

  • nym-node routes - all of the below got deprecated as they will only return legacy nym-mixnode and nym-gateway data:

    • /v1/gateways/described
    • /v1/mixnodes/described

  • Unstable Nym Nodes Routes:

    • /v1/unstable/nym-nodes/mixnodes/skimmed - due to inconsistency in behaviour (i.e. active vs all) it is now redirected to /v1/unstable/nym-nodes/mixnodes/skimmed/active and unwraps the pagination
    • /v1/unstable/nym-nodes/gateways/skimmed - due to inconsistency in behaviour (i.e. active vs all) it is now redirected to /v1/unstable/nym-nodes/entry-gateways/skimmed/all and unwraps the pagination

  • Unstable Nym Nodes Routes:
    • /v1/unstable/nym-nodes/skimmed - now works with exit parameter
    • /v1/unstable/nym-nodes/skimmed - introduced no-legacy flag to ignore legacy nym-mixnode and nym-gateway (where applicable)
    • /v1/unstable/nym-nodes/skimmed - will now return all nodes if no query parameter is provided
Mixnet Contract

Every operator has to make sure that their nodes self-described endpoint works, otherwise the node will be un-routable and thus won't get any rewards!

High Level Changes

New/Added

  • All new nodes are now bonded as Nym Nodes, even when using old BondMixnode or BondGateway messages (messages are getting translated)
  • Operators only announce nodes identity key (<ID_KEY>), host and port to the directory. Everything else is discovered via self-described endpoint
  • All Nym Nodes in the rewarded set are eligible for rewards and staking. Even if they serve one of the gateway roles. Legacy gateways can't be staked on nor get rewards.
  • All nodes, including legacy mixnodes and legacy gateways, are now uniquely identified by a monotonically increasing node_id
  • All legacy gateways are preassigned node_id during the contract migration

Removed

🔥 all concepts of node families got purged, removed, deleted, thrown into the abyss. they simply no longer exist and the world is all better for it.

Changed

  • Bunch of types got changed/renamed with some fields being added/removed/deprecated. It's be quite a lot of work to list them all here, but whenever possible and feasible, they should be cross-compatible (but not always).
  • Rewarded set is no longer just a "number". Instead it has an explicit number of all nym-node modes: mixnodes, entry and exit gateways as well as standby nodes
  • Rewarding is now based on two parameters: performance and work factor as opposed to performance and "is active" flag. However, in practice, during this transitional period, it is assumed that the work factor will be equivalent to what would have been calculated using the old "is active" flag
Transaction Messages Changes

  • BondNymNode - self-explanatory
  • UnbondNymNode - self-explanatory
  • UpdateNodeConfig - works as UpdateMixnodeConfig; it lets you change your announced host or http api port
  • MigrateMixnode - migrate your existing legacy mixnode into a Nym Node
  • MigrateGateway - migrate your exsting legacy gasteway into a Nym Node. enables staking and rewarding
  • AssignRoles - an additional step for epoch transition transactions. think of it as a replacement for AdvanceCurrentEpoch. it assigns nodes to particular roles for the given epoch

  • As mentioned, all family-related things got killed off, so the following no longer exist: CreateFamily, JoinFamily, LeaveFamily, KickFamilyMember, CreateFamilyOnBehalf, JoinFamilyOnBehalf, LeaveFamilyOnBehalf, KickFamilyMemberOnBehalf
  • UpdateActiveSetSize - the rewarded/active set are now based on the role distribution
  • AssignNodeLayer - we're no longer explicitly assigning roles to all mixnodes, instead they get assigned mixing roles
  • AdvanceCurrentEpoch - the logic for advancing the epoch/assigning active set has changed so this message was removed

v2024.12.1-aero - patch

nym-node
Binary Name:        nym-node
Build Timestamp:    2024-11-07T08:45:13.162565620Z
Build Version:      1.1.9-1
Commit SHA:         ccdee808303ffcfa8ed77176d3f629512045febb
Commit Date:        2024-11-06T16:31:30.000000000+01:00
Commit Branch:      HEAD
rustc Version:      1.82.0
rustc Channel:      stable
cargo Profile:      release

Changes

  • Fixed timeout connectivity issues with authenticator
  • Amended network allowance cap

v2024.12-aero

nym-node
Binary Name:        nym-node
Build Timestamp:    2024-10-17T08:57:52.525093253Z
Build Version:      1.1.9
Commit SHA:         d75c7eaaaf3bb7350720cf9c7657ce3f7ee6ec2e
Commit Date:        2024-10-17T08:51:39.000000000+02:00
Commit Branch:      HEAD
rustc Version:      1.81.0
rustc Channel:      stable
cargo Profile:      release

Features

  • Ensured that the cargo.toml is legible in various places; tested it on nym-node, nym-api and nymvisor.
  • Ensured that updating the cargo.toml file and restarting the given binary continues to behave as normal.
  • For the following combinations I inited the client, ran the client, stopped the client, and ran the client again:
  • Fresh client on new binary && gateway on old binary
  • Fresh client on old binary && gateway on new binary
  • Fresh client on new binary && gateway new binary
  • Existing old client on old binary & new gateway
./pg_up.sh
  • Play with the database:
docker exec -it nym-data-observatory-pg /bin/bash
psql -U youruser -d yourdb

Bugfix

Operators Guide, Tooling & Updates

Documentation Updates

Fast & Furious - WireGuard edition

Nym team started another round of load and speed testing. This time the tests are limited to Wireguard mode Gateways - to find out any weak spots for needed improvement. The load testing is happening directly on mainnet as it simulates a real user traffic which the network components must be able to handle in order.

Over past week we ran a total of three tests, with 450 clients at most. We've managed to push around 300 GB in total. Around 50% of requests failed. Over the course of those three tests, we did about 5000 requests, and bandwidth per client varies between 50Mb/s and 150Mb/s.

We already caught two bugs and fixed (opens in a new tab) it in this release.

The faster the operators upgrade to this latest release (opens in a new tab), the better. A that will allow us to do more precise testing through the nodes without the registry bug, leading to more precise specs for nym-node.

Here are the aims of these tests:

  1. Understanding of the wireguard network behavior under full load
    • How many client users can all entry gateways and exit gateways handle simultaneously?
    • How much sustained IP traffic can a subset of mainnet nodes sustain?
  2. Needed improvements of Nym Node binaries to improve the throughput on mainnet
  3. Measurement of required machine specs
    • Releasing a new spec requirements
  4. Raw data record
  5. Increase quality of Nym Nodes

Meanwhile we started to research pricing of stronger servers with unlimited bandwidth and higher (and stable) port speed, to arrive to a better understanding of needed rewards and grants to bootstrap the network before NymVPN launch.

More info about testing and tools for performance monitoring can be found in this chapter.

We would like to call out to operators to join the efforts and reach out to us if they know of solid ISPs who offer reliable dedicated services for good price or may even be interested in partnership.

Delegation Program

In October we again proceeded with our Delegation Program. 22 nodes didn't meet the program rules and got their delegation removed and 25 nodes from the que received delegation. Below is a complete list.


Delegated:

Ce6kcPckNfQsga2z645VFQYadtoTjqXrS1YXMTtNNv98
2XSCWy1vAoJRaYBJXx4KWwjU1cfoS2wNBXVQZvi8Jtdr
Bu4sUGjJqkje4vSncTH2KgrnojmfESdaYwamC6DbpJGZ
7TWEw9qQxsc8w4WhPAX6zjZ8vuNBdtP21zUVN8K26RkD
HejyqervmGTCEwi1JbRBXV5My463336huBn8ZgSpuhc3
CXcCVGiamYSwgVwaxW3mEkXkZh1sKY2TXnWjjTjxDxzA
FScLfnKUPv9wSef3R4N2pQ9ft7DiwdivLW1i65Dqfc9L
2vuZZJjyYN27fvDbhyqeGosewGWaRh6iVsFtqbJoYAR7
B9QiBsSAx7MRcTpYMs1fu9AFJurAZTPWMispHZXPbaVW
E3e2a9kXZjQXsKAfvmCf2WqwmVkiGR2LbjCwoadZgEJt
Dk4fCLM7idHPqfsUucLQtSMtYaYCLhi4T7vwvw88jG3P
9xZUp4sYWUNJesWy3MPVjh5kTorNqj3RxcFgBmYjV1xV
HK9QxPpdJfNtNpLJZHTN5M113jeBbFzTkMtPt9eouimx
ECkzyHfoiNGKyDTtbbH5HDCWa8KMGh92mtGbGHLZ3Y9n
9jQQV9vQ2mFFXywwVhACCKefjUFpyBoCU6KXNfjAEi45
6QguhCfnDPKJe8bQXg9myuPB89yYFk6R77vMhLTbipK7
4hAJJQhLTFve8FZGd28ksjavbch8STMax2rytzKmDPCV
EZLFq5HGXFKRpxu78nVjf7kuuUaKPLAbezR6mXbZrP6y
FtAAA5GMxY1Ge9wKYDrQgaSfJEUp4XvBLptBwy3GU8ap
tUiLPjz5nkPn5ZJT5ZXLPGDcZ3caQsfkMAp1epoAuSQ
4ScsM6AVowhKTMWaH98NLntKDwbu2ZMEycUk4mZiZppG
Hb34PTth6CeFziPAAEUMEjJFHWJg1dDex5QxUXKNqRBE
9ek1PMvLhpbwZe7kTMyCVY5VNqrdSPPoruFPQtbxnZyf

Undelegated due to the use of an outdated binary:

9UHXFYuMLhuugndt8xCFRydmDPFyEEUHYc72tNANEtHp
5Y86A7fUX3LYVDDeoujtAiZFudYcHJq6gw8nsp71wN7U
HYWjn6yL8y7TBPFL9bTgDm6tHgyoEQupgJuBhLLoA5EY
4JCpbdhiQFKWwhrbkNDbwcwBGZnvU4WQrF2vqQLfmZvW
2f7JaYmmrMQQMczLX32ogfP7PBHeyPKbAVNjjEsExZVd
9TW55JrsFhsMoe3Tf8LBR4bPSCX86VXyvioMmCw9tWB
AyN34XqUi5XxgjmivWG2z6TftkqAFjVV5C9zCbx8Fvp5
skNS4zNsKdbbUR9wFTJoPdmReW4NdrDEpp8512TNG4f
DztUnMKM545sdipgqhCsPNhK3YVmBbS2fp9HZgM5Jpw9
GnLmx1s7g9nH3uLRhGpaXTbQEhCSKB6YenBQWQhthSx9
GoJjAkH5hpcPYeW7JDUVfHdqgcufjwdhY2PLwBGJV3Ar
EdHVMTXpLiBbvCUnEoSPQ86pBNY1h9HtL34Q7cpNPWCy

Undelegated due to increased operation costs or profit margin:

Erw9AQ4UJCgCiAWisUWbFk9Yedm8qvW4YQqmJRrBrE5p
BVDVtmNbZRgPKU81uBkrgfj5TnhtZqQcPAwxD48jcfMd
36nmH3kawhAsNA6sxFva2HgTnQHQDbcrRefvWWbmhHvY
2831fyXRAJ88x1Pd5aW7utw7WH1XkHZEfoWhLk2foLxJ
AMDS4cib433iRstwP9mWnZ4zPqb6hm6uPF7PpvhSkpYC
DE9eEeVsuiKeVfwebg5HYsebqRUvxd7LWsT9hQUtVrTQ
FAKhiQ8nW5sAWAxks1WB8u1MAWsapToCSE3KmF9LuGRQ

Undelegated due to being blacklisted for extensive period

sjL9n9ymxfWWwkQJxXdsMkdwamXfh3AJ3vCe7rJ8RrT
E2HAJrHnk56QZDUCkcjc4i4pVEqtyuPYL5bNFYtweQuL
4PytR3tmodsvqGTKdY47yie8kmrkARQdb5Ht3Ro3ChH4

v2024.11-wedel

Binary Name:        nym-node
Build Timestamp:    2024-09-27T11:02:37.073944654Z
Build Version:      1.1.8
Commit SHA:         c3ec970a377adb25d57be5428551fada2ec55128
Commit Date:        2024-09-26T08:24:53.000000000+02:00
Commit Branch:      master
rustc Version:      1.80.1
rustc Channel:      stable
cargo Profile:      release

Features

  • Noted that the constants DEFAULT_PEER_TIMEOUT and DEFAULT_PEER_TIMEOUT_CHECK have been moved to common/wireguard-types/src/lib.rs and are now being used across modules for consistency
  • Observed that the peer_controller.rs now separates the in-memory updates from the storage sync operations to reduce system load
  • Identified that in-memory updates of peer bandwidth usage happen every DEFAULT_PEER_TIMEOUT_CHECK (every 5 seconds), while storage updates occur every 5 * DEFAULT_PEER_TIMEOUT_CHECK (every 25 seconds)

Checked System Load and Performance:

  • Monitored system resource usage (CPU, memory, I/O) during the test to assess the impact of the changes

  • Confirmed that the separation of in-memory updates and storage syncs resulted in reduced system load, particularly I/O operations, compared to previous versions where storage updates occurred more frequently

  • Ensured that the system remained responsive and no performance bottlenecks were introduced

  • Efficiency Improvement: The separation of in-memory updates and storage syncs effectively reduced unnecessary database writes, improving system efficiency without compromising data accuracy

  • Initialised new nym-client with the --latency-based-selection flag and ensured it still works as normal.
    • Checked out the release/2024.10-wedel branch containing the fix for the race condition on IP and registration structures
    • Deployed the on a controlled test environment to prevent interference

  2. Monitored Logs:

    • Enabled debug logging to capture all events
    • Monitored logs in real-time to observe the handling of concurrent registration requests
    • Checked for any error messages, warnings, or indications of race conditions

  3. Verified Client Responses:

    • Ensured that all clients received appropriate responses:
    • Successful registration with assigned IP and registration data
    • Appropriate error messages if no IPs were available or if other issues occurred
    • Confirmed that no clients were left in an inconsistent state (e.g., assigned an IP but not fully registered)

  4. Validated Normal Operation:

    • Conducted standard registration processes with individual clients to confirm that regular functionality is unaffected via nym-vpn-cli
    • Ensured that authenticated clients could communicate over the network as expected

Wireguard details are now visible at the nym-node endpoint /api/v1/gateway/client-interfaces as well as on the nym-api self-described endpoint /api/v1/gateways/described, above the existing data displaying mixnet_websocket information.

An example of what will be shown is:

 "wireguard": {
 "port": 51822,
 "public_key": "<some public key here>"
 }

Checked the following commands:

show-ticket-books # which displays the information about all ticketbooks associated to the client
import-ticket-book # which imports a normal ticketbook to the client alongside `--full` flag

On the cli, the following were added: import-coin-index-signatures, import-expiration-date-signatures and import-master-verification-key.

  1. Install with
sudo dpkg -i ./nym-repo-setup.deb
  1. Once it's installed, it should be possible to install the vpn client with
sudo apt install nym-vpnc
  1. To remove the repo, use
sudo apt remove nym-repo-setup

NOTE: removing the repo will not remove any installed nym-vpn packages

  1. Downloaded the nym-repo-setup.deb package to a Debian-based test system

  2. Installed the repository setup package using the command:

sudo dpkg -i ./nym-repo-setup.deb
  1. Verified that the GPG keyring was copied to /usr/share/keyrings/nymtech.gpg:
ls -l /usr/share/keyrings/nymtech.gpg
  1. Checked that the repository specification was added to /etc/apt/sources.list.d/nymtech.list:
cat /etc/apt/sources.list.d/nymtech.list
  1. Updated the package list:
sudo apt update
  1. Installed the VPN client meta-package:
sudo apt install nym-vpnc

  1. Confirmed that the nym-vpnc package and its dependencies (daemon and UI) were installed successfully

  2. Tested the VPN client to ensure it operates as expected

  3. Removed the repository setup package:

sudo apt remove nym-repo-setup

  1. Verified that the repository specification file /etc/apt/sources.list.d/nymtech.list was removed

  2. Ensured that the installed nym-vpnc packages remained installed and functional after removing the repo setup package

 - ci-build-upload-binaries
 - ci-build
 - ci-cargo-deny
 - ci-contracts-schema
 - ci-contracts-upload-binaries
 - ci-contracts
 - ci-docs
 - ci-nym-wallet-rust
 - ci-sdk-wasm

Bugfix

Tested running a client in mixnet mode, with a standard ticketbook, as well as a client using an imported ticketbook. The double spending bug is no longer an issue, bandwidth is consumed properly, and upon consumption of one ticket another ticket is properly obtained.

Operators Guide, Tooling & Updates

nym-node patch from release/2024.10-caramello

Binary Name:        nym-node
Build Timestamp:    2024-09-16T15:00:41.019107021Z
Build Version:      1.1.7
Commit SHA:         65c8982cab0ff3a1154966e7d61956cb42a065fc
Commit Date:        2024-09-16T15:59:34.000000000+02:00
Commit Branch:      HEAD
rustc Version:      1.81.0
rustc Channel:      stable
cargo Profile:      release

This patch fixes v202410-caramello release bug where one of the used dependencies - DefGuard (opens in a new tab), was failing.

Updating to this patched version and running nym-node --mode exit-gateway with --wireguard-enabled true should result in a smooth node start without the defguard_wireguard error, occuring to some operators before:

/home/ubuntu/.cargo/registry/src/index.crates.io-6f17d22bba15001f/defguard_wireguard_rs-0.4.2/src/netlink.rs:155: Serialized netlink packet (23240 bytes) larger than maximum size 12288: NetlinkMessage.

This release is a patch only, there are no additional features, everything else stays the same like in the latest release v202410-caramello.

v2024.10-caramello

Features

Scenario 1: Bandwidth Decreasing Continuously

  1. Started the client and noted the initial bandwidth (e.g., 1GB).
  2. Used the client and tracked bandwidth usage over time (e.g., decrease by 100MB every hour).
  3. Restarted the client after some usage.
  4. Verified the bandwidth continued from the last recorded value, not reset.

The bandwidth continued decreasing without resetting upon restart. Logs and reports correctly reflected the decreasing bandwidth.

Scenario 2: Bandwidth Reset Next Day

  1. Used the client normally until the end of the day.
  2. Suspended some clients and kept others active.
  3. Checked bandwidth at midnight.
  4. Verified that bandwidth reset to 1GB for both suspended and active clients.

Bandwidth reset to 1GB for all clients at midnight. Logs and reports correctly showed the reset.

Scenario 3: Bandwidth Reset at a Different Time (e.g., Midday)

  1. Configured the system to reset bandwidth at midday.
  2. Used the client and monitored bandwidth until midday.
  3. Kept the client connected during the reset time.
  4. Verified that bandwidth reset to 1GB live at midday.

Bandwidth reset to 1GB at midday while the client was connected. Logs and reports correctly reflected the reset.

Scenario 4: Stale Check for 3 Days

  1. Kept a client inactive for 3 days.
  2. Verified removal from the peer list after 3 days.
  3. Reconnected the client after 3 days and checked for a new private IP.
  4. Restarted a client within 3 days and verified it retained the same private IP.

The client was removed from the peer list after 3 days of inactivity. Upon re-connection after 3 days, the client received a new private IP. The client retained the same private IP when restarted within 3 days.

  • Verify that the nym-gateway binary and nym-mixnode binary commands return the error message stating to update to nym-node
  • Check that when adding the --force-run flag, it still allows the command to be run (aside from init which has been removed) and the message stating to update to nym-node is a warning now
  • Check nym-node is not affected
  • Reviewed the changes in the PR
  • Handle clients with different versions in IPR (opens in a new tab): Allow the IPR to handle clients connecting both using v6 and v7, independently. The motivation is that we want to be able to roll out a API version change gradually for VPN clients without breaking backwards compatibility. The main feature on the new v7 format that is not yet used, is that it adds signatures for connect/disconnect.

Run the same command (using same gateways deployed from this PR) on different versions of the nym-vpn-cli.

Example:

sudo -E ./nym-vpn-cli -c ../qa.env run --entry-gateway-id $entry_gateway --exit-gateway-id $exit_gateway --enable-two-hop
 
sudo -E ./nym-vpn-cli -c ../qa.env run --entry-gateway-id $entry_gateway --exit-gateway-id $exit_gateway --enable-two-hop

Bugfix

  • Building all binaries is ok
  • Running cargo fmt returns no issues

Tested updating an old nym-node version and ensuring it did not throw any errors.

#!/bin/bash
 
packages=$(cargo metadata --format-version 1 --no-deps | jq -r '.packages[].name')
 
# Loop through each package and build
for package in $packages; do
    echo "Building $package"
    cargo clean
    cargo check -p "$package"
    if [ $? -ne 0 ]; then
        echo "Build failed for $package. Stopping."
        exit 1
    fi
done

Crypto

Operators Guide, Tooling & Updates

Minimum profit margin = 20%
Maximum profit margin = 50%
Minimum operating cost = 0 NYM
Maximum operating cost = 1000 NYM

  • Nym Harbourmater (opens in a new tab) has several new functionalities:

    • Version counting graph for Gateways and Mixnodes
    • Several new columns with larger nodes performance and settings overview.
    • Top routing score now includes:
      • Wireguard registration and complete handshake test, to configure see tasklist below
      • DNS resolution check, to configure see tasklist below
      • Wireguard perfomance bigger than 0.75, to configure see tasklist below

  • New Nym Wallet (opens in a new tab) is out!

    • Vesting contract functionalities have been purged, users can only remove tokens from vesting
    • Migrating from mixnode or gateway smart contracts to a new unifying nym-node smart contract will be available soon using Nym desktop wallet, just like you are used to for bonding and node settings. After this migration all nym-nodes will be able to receive delegation and rewards. We will share a step by step guide once this migration will be deployed. No action needed now.

  • Nym API Check CLI is upgraded according to the latest API endpoints, output is cleaner and more concise.

Operators Tasks

The steps below are highly recommended for all operators and mandatory for everyone who is a part of Nym Delegation or Grant program. Deadline is Friday, September 20th, 2024.

Every nym-node should be upgraded to the latest version! Operators can test using Sandbox env during the pre-release period, then upgrade on mainnet. During the upgrade, please follow the points below before you restart the node:

nym-node

  • Make sure to fill in basic description info, into the file located at .nym/nym-nodes/<ID>/data/description.toml (all nodes)
  • Configure wireguard routing with new network_tunnel_manager.sh (opens in a new tab) following these steps (Gateways only for the time being)
  • Enable Wireguard with --wireguard-enabled true flag included in your run command (Gateways only for the time being)
    • Note: On some VPS this setup may not be enough to get the correct results as some ISPs have their own security groups setup below the individual VPS. In that case a ticket to ISP will have to be issued to open the needed settings. We are working on a template for such ticket.
  • Setup reverse proxy and WSS on nym-node (Gateways only for the time being)
  • Don't forget to restart your node - or (preferably using systemd automation) reload daemon and restart the service
  • Optional: Use nym-gateway-probe and NymVPN CLI (opens in a new tab) to test your own Gateway
  • Optional: Run the script below to measure ping speed of your Gateway and share your results in Nym Operators channel (opens in a new tab)

We made a script for pinging nymtech.net from your GWs. Can you please install it and then share the result together with your Gateway ID:

  1. Get the script onto your machine (soon on github for curl or wget):
# paste all this block as one command
cat <<'EOL' > ping_with_curl_average_for_wg_check.sh
#!/bin/bash
 
ping_with_curl_average_for_wg_check() {
    total_connect_time=0
    total_total_time=0
    iterations=5
    timeout=2
 
    for ((i=1; i<=iterations; i++)); do
        echo "ping attempt $i..."
 
        echo "curling nymtech.net to check ping response times"
        times=$(curl -I https://nymtech.net --max-time $timeout \
        -w "time_connect=%{time_connect}\ntime_total=%{time_total}" -o /dev/null -s)
 
        time_connect=$(echo "$times" | grep "time_connect" | cut -d"=" -f2)
        time_total=$(echo "$times" | grep "time_total" | cut -d"=" -f2)
 
        total_connect_time=$(echo "$total_connect_time + $time_connect" | bc)
        total_total_time=$(echo "$total_total_time + $time_total" | bc)
 
        echo "time to connect: $time_connect s"
        echo "total time: $time_total s"
    done
 
    average_connect_time=$(echo "scale=3; $total_connect_time / $iterations" | bc)
    average_total_time=$(echo "scale=3; $total_total_time / $iterations" | bc)
 
    echo "-----------------------------------"
    echo "average time to connect: $average_connect_time s"
    echo "average total time: $average_total_time s"
}
 
ping_with_curl_average_for_wg_check
EOL
  1. Make executable:
chmod +x ping_with_curl_average_for_wg_check.sh
  1. In case you don't have bc, install it:
sudo apt install bc
  1. Run:
./ping_with_curl_average_for_wg_check.sh
  1. Share results and ID key in Nym Operators channel (opens in a new tab)

THANK YOU!

validators

  • Validators need to update and prepare for ecash implementation.

Known Bugs & Undone features

  • New nym-nodes without a performance 24h history above 50% don't show routing properly on nym-gateway-probe, on Nym Harbourmaster the page may appear blank - we are working on a fix.
  • Wireguard works on IPv4 only for the time being, we are working on IPv6 implementation.
  • Harbourmaster Role column shows nym-node --mode exit-gateway as EntryGateway, we are working to fix it.
  • In rare occassions Harbourmaster shows only "panda" without the "smiley" badge even for nodes, which have T&C's accepted. We are working to fix it.
  • Sometimes nym-node running with --wireguard-enabled true gives this error on restart: Serialized netlink packet .. larger than maximum size ..
/home/ubuntu/.cargo/registry/src/index.crates.io-6f17d22bba15001f/defguard_wireguard_rs-0.4.2/src/netlink.rs:155: Serialized netlink packet (23240 bytes) larger than maximum size 12288: NetlinkMessage.

From what we found out it seems that one of our dependencies - DefGuard - is failing (opens in a new tab). Based on the reading on their fix, it seems that when node operators try to re-create a wireguard interface with too many previous peers (like on Gateway restart, with restoring from storage), there's an overflow. So their fix is to just add them one by one. To be sure that bumping the dependency version fixes the problem there's still two things we'd need to check - and your feedback would help us a lot:

  1. Did operators only encounter this error after a nym-node (Gateway) restart?
  2. Reprouce this error ourselves and see if it actually fixes our problem.

Please share your experience with us to help faster fix of this issue.

v2024.9-topdeck

Features

  • Verify that the nym-gateway binary and nym-mixnode binary commands return the _error message_ stating to update to nym-node
  • Check that when adding the --force-run flag, it still allows the command to be run (aside from init which has been removed) and the message stating to update to nym-node is a _warning_ now
  • Check nym-node is not affected
  • Review the changes in the PR

Scenario 1: Bandwidth Decreasing Continuously'

  1. Start the client and noted the initial bandwidth (e.g., 1GB).
  2. Us the client and track bandwidth usage over time (e.g., decrease by 100MB every hour).
  3. Restart the client after some usage.
  4. Verify the bandwidth continued from the last recorded value, not reset.

Notes: The bandwidth continued decreasing without resetting upon restart. Logs and reports correctly reflected the decreasing bandwidth.

Scenario 2: Bandwidth Reset Next Day'

  1. Use the client normally until the end of the day.
  2. Suspend some clients and kept others active.
  3. Check bandwidth at midnight.
  4. Verify that bandwidth reset to 1GB for both suspended and active clients.

Notes: Bandwidth reset to 1GB for all clients at midnight. Logs and reports correctly showed the reset.

Scenario 3: Bandwidth Reset at a Different Time (e.g., Midday)'

  1. Configure the system to reset bandwidth at midday.
  2. Use the client and monitored bandwidth until midday.
  3. Keep the client connected during the reset time.
  4. Verify that bandwidth reset to 1GB live at midday.

Notes: Bandwidth reset to 1GB at midday while the client was connected. Logs and reports correctly reflected the reset.

  • Handle clients with different versions in IPR (opens in a new tab): Allow the IPR to handle clients connecting both using v6 and v7, independently. The motivation is that we want to be able to roll out an API version change gradually for NymVPN clients without breaking backwards compatibility. The main feature on the new v7 format that is not yet used, is that it adds signatures for connect/disconnect.

Run the same command (using same gateways deployed from this PR) on different versions of the nym-vpn-cli.

Example:

sudo -E ./nym-vpn-cli -c ../qa.env run --entry-gateway-id $entry_gateway --exit-gateway-id $exit_gateway --enable-two-hop
 
sudo -E ./nym-vpn-cli -c ../qa.env run --entry-gateway-id $entry_gateway --exit-gateway-id $exit_gateway --enable-two-hop

Bugfix

  • Tested updating an old nym-node version and ensuring it did not throw any errors.
  • Building all binaries is ok
  • Running cargo fmt returns no issues

Operators Guide updates

  • WireGuard tunnel configuration guide for nym-node (currently Gateways functionalities). For simplicity we made a detailed step by step guide to upgrade an existing nym-node to the latest version and configure your VPS routing for WireGuard. Open by clicking on the example block below.

Prerequisites

  • Nym Node Version: You must be running the 2024.9-topdeck release branch, which operates as nym-node version 1.1.6. You can find the release here: Nym 2024.9-topdeck Release (opens in a new tab).

  • Important: Before proceeding, make sure to back up your current nym-node configuration to avoid any potential data loss or issues.

  • Download Nym Node:

  • You can download the nym-node binary directly using the following command:

curl -L https://github.com/nymtech/nym/releases/download/nym-binaries-v2024.9-topdeck/nym-node -o nym-node && chmod u+x nym-node

Step 1: Update UFW Firewall Rules

  • Warning: Enabling the firewall with UFW without allowing SSH port 22 first will lead to losing access over SSH. Make sure port 22 is allowed before proceeding with any UFW configurations.

Run the following as root or with sudo prefix:

  1. Check the current status of UFW (Uncomplicated Firewall):
ufw status
  1. Ensure that the following ports are allowed on your machine before adding the WireGuard port:
ufw allow 22/tcp    # SSH - you're in control of these ports
ufw allow 80/tcp    # HTTP
ufw allow 443/tcp   # HTTPS
ufw allow 1789/tcp  # Nym specific
ufw allow 1790/tcp  # Nym specific
ufw allow 8080/tcp  # Nym specific - nym-node-api
ufw allow 9000/tcp  # Nym Specific - clients port
ufw allow 9001/tcp  # Nym specific - wss port
ufw allow 51822/udp # WireGuard
  1. Confirm that the UFW rules have been updated:
ufw status

Step 2: Download and Prepare the Network Tunnel Manager Script

  1. Download the network_tunnel_manager.sh (opens in a new tab) script:
curl -L -o network_tunnel_manager.sh https://github.com/nymtech/nym/blob/develop/scripts/network_tunnel_manager.sh
  1. Make the script executable:
chmod u+x network_tunnel_manager.sh
  1. Apply the WireGuard IPTables rules:
./network_tunnel_manager.sh apply_iptables_rules_wg

Step 3: Update the Nym Node Service File

  1. Modify your nym-node service file to enable WireGuard. Open the file (usually located at /etc/systemd/system/nym-node.service) and update the [Service] section as follows:
[Service]
User=<YOUR_USER_NAME>
Type=simple
#Environment=RUST_LOG=debug
# CAHNGE PATH IF YOU DON'T RUN IT FROM ROOT HOME DIRECTORY
ExecStart=/root/nym-node run --mode exit-gateway --id <YOUR_NODE_LOCAL_ID> --accept-operator-terms-and-conditions --wireguard-enabled true
Restart=on-failure
RestartSec=30
StartLimitInterval=350
StartLimitBurst=10
LimitNOFILE=65536
 
[Install]
WantedBy=multi-user.target
 
# ADD OR TWEAK ANY CUSTOM SETTINGS
  1. Reload the systemd daemon to apply the changes:
systemctl daemon-reload
  1. Restart the nym-node service:
systemctl restart nym-node.service
  1. Optionally, you can check if the node is running correctly by monitoring the service logs:
journalctl -u nym-node.service -f -n 100

Step 4: Run the Network Tunnel Manager Script

Finally, run the following command to initiate our favorite routing test - run the joke through the WireGuard tunnel:

./network_tunnel_manager.sh joke_through_wg_tunnel
  • Note: Wireguard will return only IPv4 joke, not IPv6. WG IPv6 is under development. Running IPR joke through the mixnet with ./network_tunnel_manager.sh joke_through_the_mixnet should work with both IPv4 and IPv6!

For Exit Gateway:

./nym-node run --id <ID> --mode exit-gateway --public-ips "$(curl -4 https://ifconfig.me)" --hostname <HOSTNAME> --http-bind-address 0.0.0.0:8080 --mixnet-bind-address 0.0.0.0:1789 --location <LOCATION> --accept-operator-terms-and-conditions --wireguard-enabled true
 
# wireguard can be enabled from version 1.1.6 onwards

For Entry Gateway:

./nym-node run --id <ID> --mode entry-gateway --public-ips "$(curl -4 https://ifconfig.me)" --hostname <HOSTNAME> --http-bind-address 0.0.0.0:8080 --mixnet-bind-address 0.0.0.0:1789 --accept-operator-terms-and-conditions --wireguard-enabled true
 
# wireguard can be enabled from version 1.1.6 onwards
22 # SSH
123 # NTP
445 # SMB file share Windows
465 # URD for SSM
587 # SMTP
853 # DNS over TLS
1433 # databases
1521 # databases
2049 # NFS
3074 # Xbox Live
3306 # databases
5000-5005 # RTP / VoIP
5432 # databases
6543 # databases
8080 # HTTP Proxies
8767 # TeamSpeak
8883 # Secure MQ Telemetry Transport - MQTT over SSL
9053 # Tari
9339 # gaming
9443 # alternative HTTPS
9735 # Lightning
25565 # Minecraft
27000-27050 # Steam and game servers
60000-61000 # MOSH
[host]
public_ips = [
'<PUBLIC_IP>'
]
 
[mixnet]
bind_address = '0.0.0.0:1789'
 
[http]
bind_address = '0.0.0.0:8080'
 
[mixnode]
[mixnode.verloc]
bind_address = '0.0.0.0:1790'
 
[entry_gateway]
bind_address = '0.0.0.0:9000'

Tooling

v2024.8-wispa

Features

  1. Reviewed the changes in the nym-api-requests/src/models.rs file.
  2. Verified that the NymNodeDescription struct includes the new role field with a default value set by default_node_role.
  3. Checked the implementation of the default_node_role function to ensure it returns NodeRole::Inactive.
  4. Ran the updated code in the sandbox environment.
  5. Monitored the sandbox environment for any issues or errors related to the changes.

Notes (if any):

The test was successful. No issues were flagged during the testing in the sandbox environment. The new default value for NodeRole ensures backward compatibility without causing disruptions.

  1. Reviewed the changes in the gateway/src/error.rs and gateway/src/node/mod.rs files.
  2. Verified the new error enum AuthenticatorStartupFailure was added to GatewayError.
  3. Confirmed the implementation of the StartedAuthenticator struct and its usage in the start_authenticator function.
  4. Ran the updated code in the canary environment.
  5. Monitored the canary environment for any issues or errors related to the changes.
  1. Reviewed the changes in common/client-libs/validator-client/src/nyxd/cosmwasm_client/client_traits/signing_client.rs, logs.rs, types.rs, and nym-api/src/coconut/tests/mod.rs files.
  2. Verified the addition of event parsing in the relevant functions and structs.
  3. Ensured that the find_attribute function correctly parses event attributes.
  4. Ran the updated code in the sandbox environment.
  5. Broadcasted transactions on the sandbox network to test the changes.
  6. Monitored the sandbox network for any malformed responses or errors after the test chain upgrade.
  1. Reviewed the changes in common/bandwidth-controller/src/event.rs, common/bandwidth-controller/src/lib.rs, and common/client-libs/gateway-client/src/client.rs files.
  2. Verified the implementation of BandwidthStatusMessage enum for emitting status messages.
  3. Ensured GatewayClient is updated to send bandwidth status messages when connecting.
  4. Deployed the updated code on the canary environment.
  5. Connected to the gateway and checked for the emission of bandwidth status messages.
  6. Verified that the messages were correctly parsed and consumed by the application layer.
  7. Ran the VPN client to observe the parsed events.
  • Fix NR config compatibility (opens in a new tab): Recently we deleted the old statistics service provider. This fixes some issues where old configs didn't work with the latest changes.
    • Make NR able to read config with old keys in
    • Remove deleted config keys from NR template
  1. Reviewed the changes in the service-providers/network-requester/src/config/mod.rs and service-providers/network-requester/src/config/template.rs files.
  2. Ensured NetworkRequester config is able to read old keys for compatibility.
  3. Removed old and deleted config keys from the NetworkRequester template.
  4. Compiled the project to verify no issues or warnings appeared.
  5. Ran all tests to ensure that the changes did not affect the functionality.
  6. Validated that no leftover code from the old statistics service provider caused any issues.
  1. Reviewed the changes in common/http-api-client/src/user_agent.rs file.
  2. Verified the removal of the UserAgent constructor and ensured that all instances of UserAgent::new are updated accordingly.
  3. Checked the implementation of UserAgent struct using BinaryBuildInformation and BinaryBuildInformationOwned.
  4. Deployed the updated code across different environments (QA, sandbox, and canary).
  5. Ran tests to ensure that the UserAgent struct functions correctly without the constructor.
  • Add mixnodes to self describing api cache (opens in a new tab):
    • Abstracts getting the self describing info a bit
    • Adds mixnodes to the cache refresher as well
    • Adds role field to the NodeDescription struct, to be able to distinguish between mixnodes and gateways
    • Switched to using NodeStatusCache instead of ContractCache

Called the new /mixnodes/described endpoint as well as the existing /gateways/described endpoint and verified that the data returned for each was correct based on the settings that different nodes have when they are setup.

For gateway endpoint, the “role” for now does not differentiate between entry and exit gateways, this will be implemented in the future.

  1. Reviewed the changes to move and upgrade crates to the workspace.
  2. Verified the updated dependencies:
    • Moved dirs to version 4.0 in the workspace.
    • Updated the base64 dependency to use the workspace version.
    • Moved rand_chacha and x25519-dalek to the workspace.
    • Updated ed25519-dalek to use the workspace version.
    • Moved and upgraded itertools in the workspace.
    • Moved other partial dependencies to the workspace while preserving their versions.
  3. Ensured the Cargo.toml files across the project reflect these changes correctly.
  4. Compiled the entire project to check for any issues or warnings.
  5. Verified that all tests pass successfully after the changes.
  1. Reviewed the project to identify all references to nym-network-statistics.
  2. Removed all code and dependencies associated with nym-network-statistics.
  3. Ensured that no references to nym-network-statistics remain in the codebase, including comments, imports, and configuration files.
  4. Compiled the project to check for any issues or warnings.
  5. Ran all tests to ensure the removal did not affect the functionality of the project.
  1. Reviewed the project to identify all references to nym-network-statistics.
  2. Removed all code and dependencies associated with nym-network-statistics.
  3. Ensured that no references to nym-network-statistics remain in the codebase, including comments, imports, and configuration files.
  4. Compiled the project to check for any issues or warnings.
  5. Ran all tests to ensure the removal did not affect the functionality of the project.

Used the nym-vpn-cli to test this, and we can visibly see the UserAgent, no issues with the comments mentioned above.

Example of the user agent sent: nym-client/1.1.36/x86_64-unknown-linux-gnu/e18bb70

image

Connected with no problems

Bugfix

Operators Guide updates

[host]
public_ips = [
'<PUBLIC_IP>'
]
 
[mixnet]
bind_address = '0.0.0.0:1789'
 
[http]
bind_address = '0.0.0.0:8080'
 
[mixnode]
[mixnode.verloc]
bind_address = '0.0.0.0:1790'
 
[entry_gateway]
bind_address = '0.0.0.0:9000'

v2024.7-doubledecker

Features

Bugfix

Operators Guide updates

v2024.6-chomp

Features

  • Verify that the execute function correctly checks if the accept_operator_terms flag is set.
  • Test that a warning is displayed when the accept_operator_terms flag is not set.
  • Confirm that the NymNode instance is initialized with with_accepted_toc(accepted_toc) when the flag is set.
  • Apply the --accept-toc flag in the service and confirmed the change by running:
curl -s -X 'GET' 'http://18.171.251.41:8080/api/v1/auxiliary-details?output=json' -H 'accept: application/json' | jq .accepted_toc
  • Verify that the output is true.
  • Use the latest release/chomp binary with nym-node and input a dodgy ip
image
  • Validation:
image

When restarting the node it complains within the service launch file

  • New node endpoints (opens in a new tab): introduces new endpoints on nym-api (and creates scaffolding for additional ones) for providing unfiltered network topology alongside performance score of all nodes.
    • NymApiTopologyProvider got modified to use those endpoints alongside (configurable) filtering of nodes with score < 50% (like our current blacklist)
    • Old clients should work as before as no existing endpoint got removed
  • Validate that the skimmed endpoints are working, keeping in mind that they are unstable. The full-fat and semi-skimmed have not yet been implemented.
  • Check references to everything named SP and Ephemera and removed any additional references
  • Check references to everything named SP and Ephemera and removed any additional references
  • CI complains if no milestone is associated with the an issue.
  • This is a part of a bigger ticket, but initial testing has proven to shown that launching nym-nodes (entry and exit gateways) in WG enable mode to be working

QA will use this template for the other related WG tickets in this release milestone.

  • Use the latest release/chomp contracts and deploy these to QA
  • Use the nym-api to query for the results of these new contracts
 curl -X 'GET' \
   'https://qa-nym-api.qa.nymte.ch/api/v1/network/nym-contracts-detailed' \
   -H 'accept: application/json'
  • It returns a detailed view of the contracts and which branch they were built from, alongside rust versions and so forth.
image
  • This is a part of a bigger ticket, but initial testing has proven to shown that launching nym-nodes (entry and exit gateways) in WG enable mode to be working. QA will use this template for the other related WG tickets in this release milestone.

It works insofar that it compiles, we need to deploy and test this on non-mainnet before merging in

  • Purge nym-name-service contract
  • Purge nym-name-service-common
  • Purge nym-service-provider-directory contract
  • Purge nym-service-provider-directory-common
  • Remove everywhere name-service contract is used
  • Remove everywhere sp contract is used

Performed:

  • Check references to everything named SP and Ephemera and removed any additional references

Crypto

This PR contains a test failure due to the update here (opens in a new tab)

  • This is due a change in x25519-dalek from 1.1.1 to 2.
  • Crypto operations should be identical, but the byte representation has changed (sphinx clamps at creation, x25519 clamps at use). This cannot be changed in the sphinx crate without breaking changes.
  • There is a good chance that this failure doesn't impact anything else, but it has to be tested to see.
  • A mix of old and new clients with a mix of old and new mixnodes should do

Bugfix

  • Use the latest release/chomp contracts and deploy these to QA
  • If the contract was not found, the API would complain of invalid contracts, thus not starting the rest of the operations of the API (network monitor / rewarding etc)

Jun 11 16:27:34 qa-v2-nym-api bash[1352642]: 2024-06-11T16:27:34.551Z ERROR nym_api::nym_contract_cache::cache::refresher - Failed to refresh validator cache - Abci query failed with code 6 - address n14y2x8a60knc5jjfeztt84kw8x8l5pwdgnqg256v0p9v4p7t2q6eswxyusw: no such contract: unknown request

  • Use the latest release/chomp binary with nym-node and input a dodgy ip
image
  • Validation:
image
  • Verify that the establish_connection function correctly attempts to establish a connection to the gateway.
  • Test error handling for NetworkConnectionFailed by simulating a failed connection.
  • Ensure that the NetworkConnectionFailed error includes the address and source details as expected.
  • Checked that SocketState::Available is set correctly when a connection is successfully established.

warning: /home/alice/src/nym/nym/common/dkg/Cargo.toml: default-features is ignored for bls12_381, since default-features was not specified for workspace.dependencies.bls12_381, this could become a hard error in the future warning: /home/alice/src/nym/nym/common/dkg/Cargo.toml: default-features is ignored for ff, since default-features was not specified for workspace.dependencies.ff, this could become a hard error in the future warning: /home/alice/src/nym/nym/common/dkg/Cargo.toml: default-features is ignored for group, since default-features was not specified for workspace.dependencies.group, this could become a hard error in the future warning: /home/alice/src/nym/nym/common/client-libs/validator-client/Cargo.toml: default-features is ignored for bip32, since default-features was not specified for workspace.dependencies.bip32, this could become a hard error in the future warning: /home/alice/src/nym/nym/common/client-libs/validator-client/Cargo.toml: default-features is ignored for prost, since default-features was not specified for workspace.dependencies.prost, this could become a hard error in the future warning: /home/alice/src/nym/nym/common/credentials-interface/Cargo.toml: default-features is ignored for bls12_381, since default-features was not specified for workspace.dependencies.bls12_381, this could become a hard error in the future warning: /home/alice/src/nym/nym/common/credentials/Cargo.toml: default-features is ignored for bls12_381, since default-features was not specified for workspace.dependencies.bls12_381, this could become a hard error in the future warning: /home/alice/src/nym/nym/common/nymcoconut/Cargo.toml: default-features is ignored for bls12_381, since default-features was not specified for workspace.dependencies.bls12_381, this could become a hard error in the future warning: /home/alice/src/nym/nym/common/nymcoconut/Cargo.toml: default-features is ignored for ff, since default-features was not specified for workspace.dependencies.ff, this could become a hard error in the future warning: /home/alice/src/nym/nym/common/nymcoconut/Cargo.toml: default-features is ignored for group, since default-features was not specified for workspace.dependencies.group, this could become a hard error in the future.

  • This PR adds default-features = false to the workspace dependencies to fix these. An alternative way would be to remove default-features = false in the crates, but we assume these were put there for a good reason. Also we might have other crates outside of the main workspace that depends on these crates having default features disabled.
  • We also have the warning warning: profile package spec nym-wasm-sdk in profile release did not match any packages which we fix by commenting out the profile settings, since the crate is currently commented out in the workspace crate list.
  • All binaries have been built and deployed from this branch and no issues have surfaced.

Operators Guide updates

v2024.5-ragusa

Features

  • New nym-node API endpoint /api/v1/auxiliary-details: to expose any additional information. Currently it's just the location. nym-api will then query all nodes for that information and put it in the self-described endpoint.
  • New nym-node location available - use one of the three options to add this to your node config:
    1. Update the location field under [host] section of config.toml
    2. For new nodes: Initialise the node with --location flag, where they have to provide the country info. Either full country name (e.g. 'Jamaica'), two-letter alpha2 (e.g. 'JM'), three-letter alpha3 (e.g. 'JAM') or three-digit numeric-3 (e.g. '388') can be provided.
    3. For existing nodes: It's also possible to use exactly the same --location argument as above, but make sure to also provide --write-changes (or -w) flag to persist those changes!
  • Feature/unstable tested nodes endpoint (opens in a new tab): Adds new data structures (TestNode, TestRoute, PartialTestResult) to handle test results for Mixnodes and Gateways. With the inclusion of pagination to handle large API responses efficiently. Lastly, introducing a new route with the tag unstable thus meaning not to be consumed without a user risk, prefixes in endpoints with unstable, are what it says on the tin.
  • Deploy new api changes to sandbox environment
  • Ensure current operations are transactional and standed operations are working
  • Run a script to ensure that the new endpoints are working as expected with pagination
image
  • Build the project and deployed it in a test environment.
  • Manually test API endpoints for mixnode and gateway data.
  • Verify that the endpoints return the expected data and handle blacklists correctly.
  • API performance improved due to the efficient HashMap lookups
  • Data in mainnet will differ from test nets due to the increased amount of gateways and mixnodes in that environment
  • Test standard uptime routes:
curl -X 'GET' 'https://validator.nymtech.net/api/v1/status/gateway/Fo4f4SQLdoyoGkFae5TpVhRVoXCF8UiypLVGtGjujVPf/avg_uptime' -H 'accept: application/json'
  • Update a nym-api to the binary built from this branch, then restart the api
  • Check the journalctl for error messages
  • Connected via client and could not see the error messages, this is backwards compatible
  • Local testing using sdk examples:
cd <PATH_TO>/nym/sdk/rust/nym-sdk
cargo run --example simple
 
# outcome
thread 'main' panicked at sdk/rust/nym-sdk/examples/simple.rs:9:64:
called Result::unwrap() on an Err value: ClientCoreError(ValidatorClientError(NymAPIError { source: ReqwestClientError { source: reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None,
  • Run cargo autoinherit in the root directory to move dependencies to the workspace level
  • Merge the new workspace dependencies into the main list
  • Ensure no other changes were mixed in during the process
  • Verify that all feature flags for all crates remained the same as before
  • Build all the binaries from this branch to confirm successful compilation
  • Deploy the built binaries across different environments to ensure there were no issues
  • Add rustls-tls to reqwest in validator-client (opens in a new tab): An attempt to make possible to end up in a situation where use use the validator-client but without functioning TLS support. For the monorepo this is masked by cargo feature unification, but becomes a problem for outside consumers, as as been noticed in many of the vpn client implementations.
    • In validator-client: reqwest, enable rustls-tls for non-wasm32
    • In client-core: Use default features enabled for non-wasm32 and switch to webpki roots, since that's what we're using with reqwest anyway
    • In gateway-client: Switch to webpki roots, since that's what we're using with reqwest anyway

Crypto

  • Build the project to ensure no compilation errors
  • Run tests to verify the functionality of the issue_credential function
  • Execute integration tests to check the behaviour during an epoch transition.
  • Add a main.rs file in the nyxd scraper dir, underneath lib.rs, amend config.pruning_options.validate()?; to be let _ = config.pruning_options.validate(); in the mod.rs file
  • Test the different variations of pruning_options:
    • Check the default option: pruning_options: PruningOptions::default()
    • Check the nothing option: pruning_options: PruningOptions::nothing()
    • Check the custom option, example: pruning_options: PruningOptions { keep_recent: (500), interval: (10), strategy: (PruningStrategy::Custom) }
    • Check the pruning in real life for the validator rewarder
  • Validate that the database table blocks was being updated accordingly

Bugfix

  • Build the project to ensure no compilation errors
  • Run the binary with different command-line arguments to verify the CLI functionality
  • Test with and without the --no_banner flag to ensure compatibility and expected behavior
  • Verify logging setup and configuration file parsing

Operators Guide updates

  • nym-gateway-probe: A CLI tool to check in-real-time networking status of any Gateway locally.
  • Where to host your nym-node?: A list of Internet Service Providers (ISPs) by Nym Operators community. We invite all operators to add their experiences with different ISPs to strengthen the community knowledge and Nym mixnet performance.
  • Make sure you run nym-node with --wireguard-enabled false and add a location description to your config.toml, both documented in nym-node setup manual.

v2024.4-nutella

v2024.3-eclipse

API endpoints redirection

Previous endpointNew endpoint
http://<IP>:8000/statshttp://<IP>:8000/api/v1/metrics/mixing
http://<IP>:8000/hardwarehttp://<IP>:8000/api/v1/system-info
http://<IP>:8000/descriptionhttp://<IP>:8000/api/v1/description
IntroductionRelease Cycle