Freelance request: NYM -> Tor connection chaining for Whonix Workstation qube

IHavNoRamAndMustQube · June 15, 2021, 7:29pm

First of all, the only mention of freelancing I’ve found is this verbose thread, as it remains undeleted\unedited yet seen by moderator, I assume that freelance requests are allowed.

Send your price&time suggestions : nym2tor@protonmail.com
I pay in crypto of your preference (xmr?)

Details : The task is to chain packets from Whonix Workstation (aka ws-15) first through the NYM mixnet, then through Tor mixnet.
Final product I’m expecting is a detailed how-to guide, that any normal person could follow (and by that I mean copypasting commands). It will be publicly available, and maybe even will be posted in Qubes community docs (or maybe won’t, I don’t know and not affiliated with it’s admin).

Your first implementation idea may be running a NYM client inside of the Whonix Gateway qube (aka gw-15) under “clearnet” user, and just type “SocksPort localhost:1080” in torrc, but…

It lowers isolation (running 3rd party software in gw-15 is insecure cause it could be exploited to turn off tor, or tor could be exploited to turn off nym-socks5-client).
It’s impossible at the moment. as gw-15 ships with outdated version of glibc6 (2.28-10, nym-socks5-client requires 2.29) cause 15’th version is based on Debian 10 stable, Buster.
I couldn’t swap it with a new one and this practice is strongly advised against (libc is core system part).
New Debian is going to be released soon, so can expect new Whonix in… a few months?

I guess it could be roughly like that:

Building ubuntu 20 template qube as NYM is tested on it by devs; apt update && apt upgrade -y, then apt install redsocks, then wget current version of nym-socks5-client from nymtech github, chmod 777 it. All in a template.
Create an AppVM qube for running nym, check “Provides network”, run nym-socks5-client
We’re using redsocks to send tcp packets into socks5 (localhost:1080). Default redsocks incoming port is localhost:12345, so create iptables rule to redirect incoming tcp (on virtual interface from connected gw-15) to it. Also reject udp i guess.
Automate running nym-socks5-client, redsocks, iptables rules on startup after network is available and vif is known, cron “@reboot” or service. Probably do this in a template, rather than AppVM qube.
Suffer the latency, enjoy the privacy.

Well I’m not really know the stuff so I got stuck on the iptables part and realized (after a few days) I won’t make it.
If you’re need any help with NYM part (like, you’re gonna need running requester to run a nym-client) I probably can do.

P.S. If you’re aware of some other serious mixnet projects please comment them. Because I’m not.
P.P.S. If you’re using Tor, you should be interested in some violent critique on it from Dr. Neal Krawetz:
https://www.hackerfactor.com/blog/index.php?/archives/896-Tor-0day-Finding-IP-Addresses.html
https://www.hackerfactor.com/blog/index.php?/archives/868-Deanonymizing-Tor-Circuits.html
https://www.hackerfactor.com/blog/index.php?/archives/906-Tor-0day-The-Management-Vulnerability.html
P.P.P.S. Reason for using NYM → Tor and not Tor → NYM is cause NYM’s exit node (“requester”) IP is persistent and since not many people are using NYM and a lot less people will be using same requester, it’s really a fingerprint. Tor is mainstream and more common to connect to\from it’s nodes.
But yeah, if tor circuit was busted, Global Observer can see it’s coming from nym, then look up connections to NYM nodes from non-NYM residential IPs, automatically correlate timings of requests on in (mixnode) and on out (requester node) and find you (or just ddos mixnodes one by one until only the adversary mixnodes with rewritten code will be available, and find you again, like with Tor).
But that’s another story.
P.P.P.P.S. Also this (yet nonexistent) guide could help with other mixnet setup, or dVPN setup, or basically whatever supports socks5.
P.P.P.P.P.S. Don’t try to connect ws-15 directly to nym-qube, it won’t work cause ws-15 is tweaked to get connection only from gw-15. Yea if you know iptables you probably can change that.

hackerfactor · June 16, 2021, 2:17am

“violent”? I didn’t think I was violent…

deeplow · June 16, 2021, 12:00pm

This is better suited for the whonix forum: (not sure what their policy on that is, though)

IHavNoRamAndMustQube · June 16, 2021, 4:32pm

I don’t think it’s related to Whonix more than to Qubes: every draft-guide task happens in ubuntu-20, and all you do with Whonix Gateway is connect it to nym-qube (u20).

Sven · June 17, 2021, 3:42am

My impression is that this is indeed a Qubes OS related request. It involves multiple (proxy) qubes and related firewall configurations. An advanced Qubes OS config task.

I’d also be interested in reading and learning from the result.

icequbes1 · June 17, 2021, 4:01pm

You’re essentially doing what Whonix already does. Just reference their implementation. From a quick look,

You want your own qubes-network service instead of the default, so that it doesn’t forward or perform NAT
You’d want to put firewall rules in /etc/qubes/ip[6]tables.rules
You’d probably want to disable the default qubes-firewall service

See the iptables sample at GitHub - darkk/redsocks: transparent TCP-to-proxy redirector for how to do a transparent redirect, just make sure the input interface is vif+.

IHavNoRamAndMustQube · June 17, 2021, 6:02pm

You’re essentially doing what Whonix already does. Just reference their implementation.
You want your own qubes-network service

Cool but I’m not technical enough.

Well I used rules like in readme you’ve linked (many varied tries, interface (like “vif14.0”) was used as a rule match criteria), they just didn’t work for some reason.
Maybe that reason is below (I wasn’t even aware of this service…):

You’d probably want to disable the default qubes-firewall service

Gotta try in 24h repeat all with sudo service qubes-firewall stop, thanks (but I don’t feel like I’ll succeed)!

Sven · June 17, 2021, 9:58pm

Thanks BTW for the hackerfactor links… what a rabbit hole. I got hours of reading to do.

icequbes1 · June 18, 2021, 3:00am

Why do people love saying this!

At minimum you’d need:

# Redirect all TCP packets coming in from any vif##.0 interface to host_vif_ip:12345 
iptables -t nat -I PREROUTING -i vif+ -p tcp -j REDIRECT --to-ports 12345

# Ensure host_vif_ip:12345 accepts input
iptables -I INPUT -i vif+ -p tcp -m tcp --dport 12345 -j ACCEPT

# No routing, just mitm'ing
echo 0 > /proc/sys/net/ipv4/ip_forward
echo 0 > /proc/sys/net/ipv6/conf/all/forwarding

# And just in-case
iptables -P FORWARD DROP
iptables -F FORWARD

Then run redsocks on 0.0.0.0:12345 [not localhost, as a REDIRECT doesn’t hop interfaces] feed it to your upstream SOCKS server on localhost:1080.

But…you need to account for leaks, you need to deal with DNS, the rules could be refined further such as only new SYNs get redirected, etcetera; stuff the Whonix team has figured out and probably tested.

And then you’d want to make sure your “nym” ProxyVM only allows nym traffic out, etcetera, etcetera. Probably doing this by running whatever daemon it is under a designated uid/gid and doing a -m owner match, and disallowing all other OUTPUT out of the upstream interface (eth0).

The recommendation to disable qubes-firewall is only because it might interfere. Aside from setting up initial user-supplied rules from /rw/config, it dynamically modifies iptables/nft rulesets to ensure packets are forwarded. For your purposes, forwarding is bad [as well as NAT]. Don’t expect any Qube Settings-configured Firewall rules to work.

This is why I say just take the time to understand what Whonix has done and replicate it. Find out what the TransparentPort is in the torrc and see what corresponding stuff iptables has for it. I don’t have much Whonix experience but I presume this is stuff they’ve had to have figured out already.

IHavNoRamAndMustQube · June 19, 2021, 3:22pm

I tried yesterday with your iptables rules (and disabled qubes-firewall service), aaaand it worked! Many thanks!

NYM adds a lot of latency, especially when chaining NYM → Tor. Request could take about 30 seconds and more. I’ll reply when got this issue solved.

NYM → Tor (ws-15 → gw-15 → nym-qube) works, because Tor does not require DNS to function. You can send requests to domains from Whonix Workstation.
But, just NYM-proxying (say, ubuntu-20 → nym-qube) works only with IPs, not domains.

On DNS: Tor supports proxying DNS queries alone:

DNSPort [ address :] port | auto [ isolation flags ]

If non-zero, open this port to listen for UDP DNS requests, and resolve them anonymously. This port only handles A, AAAA, and PTR requests—it doesn’t handle arbitrary DNS request types. Set the port to “auto” to have Tor pick a port for you. This directive can be specified multiple times to bind to multiple addresses/ports. See SocksPort for an explanation of isolation flags. (Default: 0)

And also socks5 protocol supports DNS too. Whonix Gateway specifies in /etc/torrc.d/65_gateway.conf DnsPort as 5400

###########################
# Gateway Trans/Dns-Port #
###########################

# TransPort and DnsPort are not enabled in gateway firewall by default.
##
# (comment mirrored from /usr/bin/whonix_firewall)
# Transparent Proxy Ports for Whonix-Gateway
# TRANS_PORT_GATEWAY=“9041”
# DNS_PORT_GATEWAY=“5400”
TransPort 127.0.0.1:9041
DnsPort 127.0.0.1:5400

And in iptables there’s related rule:

user@host:~$ sudo iptables -L | grep 5400
ACCEPT udp – anywhere localhost owner UID match tinyproxy ctstate NEW udp dpt:5400

Mmm tinyproxy?

Tinyproxy is a light-weight HTTP/HTTPS proxy daemon for POSIX operating systems.

In /etc/tinyproxy/tinyproxy.conf it’s listening port specified as 8888. But there’s no such port in iptables mentioned, but tinyproxy’s UID rules related are:

user@host:~$ sudo iptables -v -L | grep tinyproxy
0 0 ACCEPT udp – any any anywhere localhost owner UID match tinyproxy ctstate NEW udp dpt:5400
0 0 ACCEPT tcp – any any anywhere localhost owner UID match tinyproxy ctstate NEW tcp dpt:9041
tinyproxy ctstate NEW tcp dpt:9041

And there’s no packets and no bytes passed on these rules (despite I used nslookup in a Workstation connected). Confusing (Whonix docs says that unconfigured for isolation (isolation means using different SocksPort for each app) apps are using Tor’s DNSPort).

TODO basically:

route DNS requests in socks5 tunnell
allow outbound traffic only for specific UID\GID (of user that will run proxy client)
unknown important stuff implemented in Whonix Gateway already
test if connection leaks when proxy client or redsocks fails for some reason?

Also, Whonix team doesn’t seem to document technological solutions they’re using. For example, if you’re searching tinyproxy in their Wiki\docs, there’s no mention of how exactly it used. Some results from the dev forum, some from the phabricator, but still really unclear imo. Tinyproxy is just an example of such obscurity.

Some articles on DNS, no tech details of basic DNS-routing implementation though:

IHavNoRamAndMustQube · July 23, 2021, 11:30am

Yesterday new NYM’s testnet was released, and it shows great (comparing to old one) latencies:
Avg latency through nym (https://ifconfig.me): 3.58 (1.68-8.18)
Avg latency through nym->tor (https://ifconfig.me): 6.873 (3.56-11.18)
(in telegram I’ve got ~1-2 sec latencies and\but any photo loads in minutes)

Time to load youtube main page, 2nd time (cache enabled), nym: 32
Time to load youtube main page, 2nd time (cache enabled), nym->tor: 34
Firefox 89, no addons.

It kinda suitable for IM and browsing right now (every latency is ok if you’re paranoid enough), so I made some scripts for quick setup and smooth running. Redsocks have some UDP and DNS mechanics, allowing to pass it through TCP proxy - but Tor doesn’t need DNS and uses only TCP, so I don’t mention those mechanisms in config (if you want use just NYM without tor - proxychains-ng\software-settings, I guess). There’s 2 requester instances in this script (one is mine, one is maxnym’s (git, twi)), if they both are offline - you’ll have to spin up your own. Also, do not download big files, currently it can kill requester-server app. Ok let’s begin.

Use ubuntu 20.xx template, because NYM’s client requires modern rust version, which is incompatible with current debian-10 and whonix-workstation-15. Then…

In template, run sudo apt install libevent-dev -y, then create new appVM with “Provides Network” mark checked. In it:

wget https://github.com/nymtech/nym/releases/download/v0.11.0/nym-socks5-client_linux_x86_64
git clone https://github.com/semigodking/redsocks
cd redsocks && make DISABLE_SHADOWSOCKS=true ENABLE_HTTPS_PROXY=false
cd ../

Create running_nym.sh file with such contents:

#!/bin/bash
#nym2tor_guy

sleep 1

recreate_nym(){
	echo recreate_nym
	providers=("813EFPEWU5G9QWCvgktVJthNbfezwpwYfMCxeYLThWcX.F9ZzKMrKx6gREik17M45yKyprELwquCniHimwwKEKL6f@8yGFbT5feDpPmH66TveVjonpUn3tpvjobdvEWRbsTH9i" "AFB7kzofcDSJ1feEJsfHE5uxq4wJecLz8MkWVywAzMCu.DZex1uSmS5iLxbc1zR96T1dDs9Wmi8ko7qjX4ACCTYQR@8yGFbT5feDpPmH66TveVjonpUn3tpvjobdvEWRbsTH9i")
	provider=${providers[ $[$RANDOM % ${#providers[@]}] ]}
	echo "choosen provider: $provider"
	pkill nym-socks5
	randid=$RANDOM
	echo '' > nym_logs
	/home/user/nym-socks5-client_linux_x86_64 init --id my-socks5-client-$randid --provider $provider > nym_logs
	/home/user/nym-socks5-client_linux_x86_64 run --id my-socks5-client-$randid > nym_logs &
}

echo '' > nym_logs
while true; do
	sleep 1
	if [[ $(stat --printf="%s" nym_logs) -gt 1 ]]; then
		keywords=("panicked" "No valid topology detected" "Aborted" "Failed to load config")
		#keep calm and detect valid topology (or die)
		for i in "${keywords[@]}"; do
			if [[ $(grep "$i" nym_logs) != "" ]]; then
			 	recreate_nym
			 	break
			fi
		done
	else
		recreate_nym
	fi
done

and running_redsocks2.sh

#!/bin/bash
#nym2tor_guy

sleep 1

# thx for iptables rules to icequbes1
# Redirect all TCP packets coming in from any vif##.0 interface to host_vif_ip:9000 
iptables -t nat -I PREROUTING -i vif+ -p tcp -j REDIRECT --to-ports 9000

# Ensure host_vif_ip:9000 accepts input
iptables -I INPUT -i vif+ -p tcp -m tcp --dport 9000 -j ACCEPT

# No routing, just mitm'ing
echo 0 > /proc/sys/net/ipv4/ip_forward
echo 0 > /proc/sys/net/ipv6/conf/all/forwarding

# And just in-case
iptables -P FORWARD DROP
iptables -F FORWARD

pkill redsocks2
/home/user/redsocks/redsocks2 -c /home/user/redsocks2_conf

then make them executable
chmod 700 *.sh

and create redsocks2 config redsocks2_conf with such contents:

base {
        log_debug = off;
        log_info = on;
        log = stderr;
        daemon = off;
        redirector = iptables;
        reuseport = off;
}
redsocks {
        bind = "0.0.0.0:9000";
        relay = "127.0.0.1:1080";
        type = socks5;
        autoproxy = 0;
}

Now we need to decide: will you use it for Whonix Workstation or for some other OS.
WS-15 is able to get connection only from gw-15, so run both scripts and connect your sys-whonix netVM to new sys-nym netVM (which is connected to sys-firewall).
But with, say, fedora\debian\ubuntu\etc you can just run Tor in same “sys-nym” netVM qube (but it lowers isolation), just add in torrc Socks5Port 127.0.0.1:1080 and replace in redsocks2_conf relay = "127.0.0.1:1080" with relay = "127.0.0.1:9050".
Final step is to run this scripts on boot, simply append lines

/home/user/running_nym.sh &
/home/user/running_redsocks2.sh &

to /rw/config/rc.local.

There’s a lot of improvement room in those scripts, I guess… So feel free to contribute or report something unusual. Maybe while true thing will go wrong as log file emerges, maybe nym’s socks5 client will fail with new keywords, so on.
I actively waited for this to happen for a long time now. I think a year. In despair I even wrote working script, chaining public proxies together, lol (probably nice for anonymity cause a lot of people and scripts are using them, but again latency).

IHavNoRamAndMustQube · August 3, 2021, 8:39am

running_nym.sh script update, with more stop-words and restarting loop (right now there’s troubles with obtaining topology). Also new requester. If you have constant “failed to get config” issues, then rm -rf ~/.nym/socks5-clients/*. How to view logs: watch -n0.1 'tail nym_logs'.

#!/bin/bash
#nym2tor_guy

sleep 1

recreate_nym(){
	while [[ $(grep 'failed to lookup address information' nym_logs) == "" ]]; do
		echo recreate_nym
		providers=("4YDNJGVBK6A33YwGkbT2Qvun4QzsDSPyJLoGxEBRU3AS.AW2MUSQBry6FH6bZLf875BUawN66bmwppwoVJ7yzoifb@7rwqR6XVwCEAph1pjxS3qtS1Rt9RRSW2LXHrns5X7ARW" "4YDNJGVBK6A33YwGkbT2Qvun4QzsDSPyJLoGxEBRU3AS.AW2MUSQBry6FH6bZLf875BUawN66bmwppwoVJ7yzoifb@7rwqR6XVwCEAph1pjxS3qtS1Rt9RRSW2LXHrns5X7ARW")
		provider=${providers[ $[$RANDOM % ${#providers[@]}] ]}
		echo "choosen provider: $provider"
		pkill nym-socks5
		randid=$RANDOM
		echo '' > nym_logs
		/home/user/nym-socks5-client_linux_x86_64 init --id my-socks5-client-$randid --provider $provider &
		sleep 2
		/home/user/nym-socks5-client_linux_x86_64 run --id my-socks5-client-$randid 2> nym_logs 1> nym_logs &
		sleep 5
	done
}

echo '' > nym_logs
while true; do
	sleep 1
	if [[ $(stat --printf="%s" nym_logs) -gt 1 ]]; then
		keywords=("panicked" "No valid topology detected" "Aborted" "Failed to load config" "the network topology is invalid")
		#keep calm and detect valid topology (or die)
		for i in "${keywords[@]}"; do
			if [[ $(grep "$i" nym_logs) != "" ]]; then
			 	recreate_nym
			 	break
			fi
		done
	else
		recreate_nym
	fi
done

Sname · January 30, 2022, 9:41am

Did you manage to set it properly ?
Do you want me to do a Salt config file and create sys-nym as network provider ?

will do it for free and just for fun because I want something rather than VPN.