Multi-network Ethereum dApp in Raku

# Multi-network Ethereum dApp in Raku

8th August 2021, The First 🦋 Raku Conference / [@knarkhov](https://narkhov.pro)

---

# Ethereum networks

.fullheight[![Networks list](https://gitlab.com/pheix-presentations/trc2021/-/raw/main/html/assets/networks.jpg)]

---

# Take more than one!

Ethereum blockchain could be much more decentralized if we will use the mix of main net, public test nets and private nets.

Such method makes it possible to increase the resilience of our distributed application against network attacks or their natural degradation.

---

# A little comparison

[Rikenby](https://www.rinkeby.io/#stats) testnet current stats:

.fullwidth[![Network rikenby](https://gitlab.com/pheix-presentations/trc2021/-/raw/main/html/assets/net-rikenby-stats.jpg)]

[Görli](https://stats.goerli.net/) testnet current stats:

.fullwidth[![Network goerli](https://gitlab.com/pheix-presentations/trc2021/-/raw/main/html/assets/net-goerli-stats.jpg)]

---

# From private to public

By the way, when we use the mix of private and public nets, we also get the additional data access level: sensitive data is stored only on private net and open data — on public net. This approach is quite good for third-party audit or tracing.

To manage data in this heterogeneous environment we need data relays (or routers) providing transparent data flows from one nets to another.

---

# Raku and its ecosystem as the platform

Raku is very friendly for Ethereum blockchain: actually, everything for quick start we get from [Rakudo Star](https://rakudo.org/star) bundle: UTF8, big numbers, JSON and UserAgent tools.

One dependency to be installed: **Net::Ethereum** module. This is a Raku interface for interacting with the Ethereum blockchain and its ecosystem via HTTP/HTTPS JSON RPC API. Module is available from: https://gitlab.com/pheix/net-ethereum-perl6.

---

# Pheix

Modern [Pheix](http://pheix.org) is the CMS for Web3.0 oriented projects. The system basics are: decentralized modules, security and storing the content in different target storages: regular databases, private and public ethereum networks.

.ethelia[Ethelia project]: [Early alfa draft of service specification](https://gitlab.com/pheix-io/service/-/tree/requirements/docs).

---

class: center, middle dark-bg light-text add-text-shadow
background-image: url(https://gitlab.com/pheix-presentations/trc2021/-/raw/main/html/assets/abstract-multi-network.jpg)
background-size: cover

# Abstract multi network relay/gateway

---
class: middle add-space-text

# Wallets, faucets and balances

**Ethereum wallets** are applications that let you interact with your Ethereum account. Think of it like an internet banking app. Most wallet products will let you generate an **Ethereum account**.

An Ethereum account is an entity that can send transactions and has a balance. Ethereum currency units are **Ether**, **Gwei** and **Wei**.

Ether on testnets has no real value, therefore, there are no markets for testnet Ether. Since you need Ether to actually interact with Ethereum, most people get testnet Ether from **faucets**.

---
class: middle add-space-text

# Shared and mocked accounts

A **shared** (or **mocked**) account is an idiom used to define an account (address) which is used simultaneously in both public and private Ethereum networks. Usage of mocked account provides:

* transaction signing on private node;
* restriction of transaction flow;
* end-to-end transactions filtering.

---

# Local vs Cloud

Access to public network could be granted by;

* **Local** (private) node (local node setup [HOW-TO](https://ethereum.org/en/developers/docs/nodes-and-clients/run-a-node/));
* **Cloud** (public) node: [infura.io](https://infura.io/), [zmok.io](https://zmok.io/), [pokt.network](https://www.pokt.network/).

The one of the main advantages of local node usage is ability to inherit authentication features stack from node software. Obvious disadvantage — node maintenance, setup & configuration, update and health monitoring.

---

# Stay cloud?!

Actually **cloud nodes** have other restrictions:

* daily requests limitations;
* restricted access to node logs;
* disabled authentication;
* speed and bandwidth;
* third-party dependency;

---

# Transaction signing problem

We can sign transaction with sequental requests:

* `personal_unlockAccount` and `eth_sendtransaction` ([1](https://geth.ethereum.org/docs/rpc/ns-personal#personal_unlockaccount), [2](https://eth.wiki/json-rpc/API#eth_sendtransaction)) to private Ethereum node;
* `personal_unlockAccount`, `signTransaction` and `sendRawTransaction` ([1](https://geth.ethereum.org/docs/rpc/ns-personal#personal_unlockaccount), [2](https://eth.wiki/json-rpc/API#eth_signtransaction) and [3](https://eth.wiki/json-rpc/API#eth_sendrawtransaction)) to private Ethereum node;
* `personal_unlockAccount`, `signTransaction` to private Ethereum node with mocked/shared account and final request `sendRawTransaction` to public Ethereum node;

---

# Manual signing

Transaction could be signed manually with [rlp](https://eth.wiki/fundamentals/rlp), [ECC](https://en.wikipedia.org/wiki/Elliptic-curve_cryptography), [Keccak-256](https://en.wikipedia.org/wiki/SHA-3) and [Secp256k1](https://en.bitcoin.it/wiki/Secp256k1) + final request `sendRawTransaction` to public Ethereum node.

I consider the manual transaction signing as extended feature of `Net::Ethereum` module. Currently `OpenSSL` module provides almost everything we need to implement this feature, one thing to be sorted out is [rlp](https://eth.wiki/fundamentals/rlp) and I think it could be done in background as a new Raku module.

---

# Mocked accounts deep dive

If we want to commit transactions to public Ethereum network without access (setup, maintenance) to full public network node, **we should use mocked/shared accounts and sign transactions locally**.

In Ethereum we have private key assigned to account. This key is secret, so if someone will get it — he will get the access to account funds and easily do withdrawal.

.disclaimer[Any activity around your private key has potential risk, do not share the keys or store them on unsecure devices and services — you can loose all your funds!]

---

# Keystore file or pure private key

To be honest, no matter. if you have original `Keystore file`, you can add new account to Ethereum node on fly.

You can easily generate `Keystore` file from private key, by the way you can set up new password phrase — this is the method how to reset password for `Keystore file`. I will use `node.js` with a few pre-installed packages ([1](https://www.npmjs.com/package/fs), [2](https://www.npmjs.com/package/ethereumjs-wallet)) for this.

---

# Generating keystore file with node.js

```javascript
const fs       = require("fs");
const wallet   = require("ethereumjs-wallet").default;
const password = 'rinkeby-node';

const pk = new Buffer.from(
    '4895d93fc350d40b2d7c181b4e3b2a01fa7f2dacf1bc4fa7a8a68782fd3adc09',
    'hex'
);

const account = wallet.fromPrivateKey(pk);

account.toV3(password)
    .then(value => {
        const addr = account.getAddress().toString('hex')
        const file =
            `UTC--${new Date().toISOString().replace(/[:]/g, '-')}--${addr}.json`
        fs.writeFileSync(file, JSON.stringify(value))
    });
```

---

# Extract private key from keystore file

By the way, you can extract private key from `Keystore file`. Simple way to use `node.js` as well, but more complex and more curious way to do it with **Python**: https://gitlab.com/pheix-io/service/-/blob/requirements/docs/auth.md#decrypt-private-key-with-python3.

This is the true dive to Ethereum crypto algorythms 🤘🤘🤘

---

# JSON RPC API

Mocked/shared could be created via HTTP(S) JSON RPC API:

* [method](https://geth.ethereum.org/docs/rpc/ns-personal#personal_importrawkey) `personal_importRawKey` (Geth);
* [method](http://cw.hubwiz.com/card/c/parity-rpc-api/1/7/10/) `parity_newAccountFromSecret` (OpenEthereum);
* [method](http://cw.hubwiz.com/card/c/parity-rpc-api/1/9/11/) `parity_newAccountFromWallet` (OpenEthereum).

You can check the status of newly created account with JSON RPC API method `personal_listAccounts` both in Geth and OpenEthereum.

---

class: center, middle tab-header
background-image: url(https://gitlab.com/pheix-presentations/trc2021/-/raw/main/html/assets/hybrid-model.jpg)
background-size: cover

# .tab-header[Hybrid model with local signer]

---

# Restrictions

The trivial transaction signing hybrid model has the next restrictions:

* dApp should use only public Ethereum network;
* we should have live local Ethereum node.

To make things easier we can use public Docker container https://gitlab.com/pheix-pool/docker-ethereum for testing. This container has all auxiliary tools for quick start, BTW there is a pre-deployed repository of local private PoA network with 2 nodes.

---

# Start it up!

```sh
docker start ethereum-dev && docker attach ethereum-dev
cd /ethereum-local-network/utils
PUBLICTESTNET=goerli ./run-geth.sh -s 1
```

To check the node status from host, try:

```bash
curl --data '{"method":"web3_clientVersion","params":[],"id":1,"jsonrpc":"2.0"}' -H "Content-Type: application/json" -X POST http://127.0.0.1:8540 | jq '.result'
```

Finally we should add mocked/shared account to node:

```bash
curl --data '{"method":"personal_importRawKey","params":["86eaa755826638cb0297cf1ad6d2fe2aa7a12b4a072ad2ff704636a3c01cb040","node1"]}' -H "Content-Type: application/json" -X POST http://127.0.0.1:8540
```

---

# Multi-network Ethereum dApp

The general assumption is that the application uses the **Hybrid model with local signer**. In this case the dApp should:

* serve the pairs of Ethereum nodes — in every pair we have public node and private node, if want to commit the transaction to public node, it should be previously signed on private node;
* provide fliexible config tools to quick setup the pairs.

---

# Features depended on Raku core

From the point of information security **HTTP/HTTPS JSON RPC API** should be disabled on our private Ethereum node. We should use **Unix domain sockets** instead.

In Raku we had no **Unix domain sockets** support for a long time and we had to do some tricky things like using Perl5 `IO::Socket::UNIX` module with Raku `Inline::Perl5`. Finally the **Unix domain sockets** were [added](https://gitlab.com/pheix-pool/core-perl6/-/issues/138) over TCP to `IO::Socket::INET`.

---

# Minimal patches for Net::Ethereum

**Hybrid model with local signer** requires the updates at `Net::Ethereum` module. The next requests should be added:

* `eth_signTransaction()`;
* `eth_sendRawTransaction()`;
* `eth_getTransactionCount()`.

Here's the list of minimal changes at `Net::Ethereum`: [@a67a84](https://gitlab.com/pheix/net-ethereum-perl6/-/commit/a67a84cb1bf3684059d7c5e0cb9861858d665069)

---

# Minimal patches for Pheix

`Pheix` should be updated as well:

* add config tools for signers;
* add support for node pairs;
* update `write_blockchain()` method.

Here's the list of minimal changes at `Pheix`: [@28eab83](https://gitlab.com/pheix-pool/core-perl6/-/commit/28eab83acdb5727679a4c7ff7e3ac5d3a8f4bb34)

---

# Subscription service: sync or async

Both synchronous and asynchronous approaches can be used to relay data on a public network.

.ethelia[Sync] case is trivial: the application updates data in a public network while updating data in a private PoA blockchain.

.ethelia[ASync] approach assumes that the application runs exclusively on a private PoA blockchain. At the same time, an auxiliary service (daemon) is working in the background and has been subscribed to events (all or selectively, depending on the settings) generated by the smart contract.

---

# Raku lang specifics again

To implement asyc subscription to events we need:

* unix domain sockets support;
* multithreading for parallel events processing.

Unix domain sockets are implemented in `IO::Socket::INET` and are available without installing any additional modules.

Both low-level (**Threads**) and high-level (**Promises**, **Supplies**) tools are available out of the box in Raku.

Also in the Raku ecosystem there is a handy `UNIX::Daemonize` module that allows you to start daemons and provide additional tools such as lockfiles and controls.

---

# Subscription service prototype

The prototype consist of starter bash script, simple Raku helper script for private PoA node requesting and public node communication Raku script that uses a few **Pheix** submodules and methods.

Starter bash script has extremely simple logic, it works in infinitive duty loop with a few stages:

* subscribe to Geth client events;
* filter events by metadata;
* execute the helper scripts on event metadata matching.

---

# Daemonize and port forward

Service could be run in daemonized mode with `nohup`.

Also we should forward Geth unix socket to TPC one with `socat` — just to be able to use `nc` from starter bash script. This should be done on server with private Ethereum node:

```bash
socat TCP-LISTEN:12345 UNIX-CONNECT:/ethereum-local-network/geth.local/geth-node1/geth.ipc
```

---

# Get remote node

The remote node JSON RPC API port should be forwarded to localhost with:

```bash
socat TCP-LISTEN:8550,reuseaddr,fork TCP:172.16.2.78:40881
```

Service prototype sources are available at: https://gitlab.com/pheix-io/service/-/tree/requirements/docs/files/subscription.

More details on subscription: https://gitlab.com/pheix-io/service/-/blob/requirements/docs/subscription.md

---

class: center, middle tab-header
background-image: url(https://gitlab.com/pheix-presentations/trc2021/-/raw/main/html/assets/make-it-live.jpg)
background-size: cover

# .tab-header[Time for live demo!]

---

# Perspectives: Pheix Release Candidate 1

The important `Pheix` updates are related to subscription service. The gems are:

* Do pure Raku implementation;
* Support events handling in multile threads;
* Optimize relay time;
* Add JSON REST API for service monitoring;
* Add extended logging;
* Try to add [fixes](https://github.com/hipek8/p6-UNIX-Daemonize/issues/6) to `Unix::Daemonize` and use it for handy service daemonization.

RC1 development process trace: https://gitlab.com/pheix-pool/core-perl6/-/milestones/3.

---

# Sensitive data tracking system

.croppedwidth90[![Heterogeneous](https://gitlab.com/pheix-presentations/trc2021/-/raw/main/html/assets/heterogeneous.gif)]

---

# Open call

I would like to invite you to .ethelia[Pheix] and .ethelia[Ethelia] dev process — code review, forks and merge requests are very welcome:

https://pheix.org

https://gitlab.com/pheix-pool/core-perl6

https://gitlab.com/pheix/net-ethereum-perl6/

---

# Thank for your attention!

.contacts[
https://narkhov.pro<br>
https://linkedin.com/in/knarkhov<br>
https://gitlab.com/pheix<br>
]