On one of our AWS EC2 servers running a Next.js development environment, we discovered an active exploitation of the React2Shell vulnerability (CVE-2025-55182 / CVE-2025-66478). The attacker successfully installed cryptocurrency mining software with persistent cron jobs. This incident report details our discovery, response, and recommendations for the broader community.

Incident Timeline & Discovery

While investigating unusual server performance, we identified unauthorized mining activity connected to a Monero mining pool. Analysis revealed:

• Mining Pool: MoneroOcean (moneroocean.stream)

• Attacker's Wallet Address: 42ZUiZsAoLeLjcaYK6rYQw1jyhadoB5kk5jXY5FJKoCkWzm964rqqBuGugSjqQZDB2B3JgE1YTGcHLEVN4nHNspH1UZX4nR

• Scope: 33+ active workers initially detected across multiple cloud providers (AWS, Azure, GCP)

• Current Status: Worker count has grown to 43+ as of this writing

This indicates a widespread, ongoing campaign targeting vulnerable servers across multiple cloud platforms.

Technical Background: The React2Shell Vulnerability

What is React2Shell?

React2Shell (CVE-2025-55182) is a critical remote code execution (RCE) vulnerability with a CVSS score of 10.0 - the maximum severity rating. It affects:

• React Server Components (RSC) in React 19.x

• Next.js 15.x and 16.x using App Router

• Next.js 14.3.0-canary.77 and later canary releases

How Does It Work?

The vulnerability stems from unsafe deserialization in the React Server Components "Flight" protocol. An attacker can:

1. Send a single crafted HTTP POST request to any exposed Next.js App Router endpoint

2. Exploit the protocol's insecure handling of RSC payloads

3. Execute arbitrary code on the server with near-100% reliability

4. Establish persistence through cron jobs and backdoors

Critical Note: Default configurations are vulnerable - no code changes by developers are required for exploitation. A standard Next.js app created with create-next-app and built for production can be exploited immediately.

Attack Vector & Exploitation

What We Observed

The attacker leveraged CVE-2025-55182 to:

1. Initial Access: Exploited the vulnerable Next.js application endpoint

2. Payload Delivery: Deployed cryptocurrency mining binaries

3. Persistence: Created cron jobs to ensure the mining process always runs

4. Evasion: Implemented mechanisms to restart the process if terminated

5. Likely Backdoor: Given the sophistication, backdoors and additional access methods were likely established

Attempted Remediation Failures

Our initial remediation attempts were unsuccessful:

# Attempted process termination - FAILED
kill -9 <mining_process_pid>
# Process automatically restarted

# Attempted cron job removal - FAILED  
crontab -e  # Removed malicious entries
# Miner continued running through alternative persistence mechanisms

# Attempted package updates - FAILED
npm install next@latest
# Downloads blocked/intercepted by attacker's modifications

The attacker's persistence mechanisms and potential system-level access prevented clean remediation.

Our Response & Recovery

Decision: Complete Server Replacement

Given the severity and persistence of the compromise, we implemented a full server replacement strategy:

1. Provisioned New EC2 Instance: Launched a fresh instance with updated configurations

2. Code Migration: Deployed application code to the new instance with patched dependencies

3. Security Hardening: Implemented additional security controls before going live

4. Old Instance Termination: Completely destroyed the compromised server

Why Full Replacement?

• Unknown Compromise Depth: No way to guarantee removal of all backdoors

• Potential Credential Theft: SSH keys, environment variables, and secrets may have been exfiltrated

• Secondary Persistence: Attacker may have established multiple persistence mechanisms

• Cost-Benefit: Rebuilding was faster and more secure than forensic cleanup

Critical Consideration: Our server only hosted frontend applications with no databases or sensitive state. Had this contained databases, user data, or cryptographic keys, the impact would have been catastrophic.

Recommended Actions - Immediate

1. Check for Vulnerable Versions

Run the official scanner:

npx fix-react2shell-next

This will:

• Scan all package.json files (supports monorepos)

• Identify vulnerable versions of React and Next.js

• Recommend specific patched versions

• Optionally apply fixes automatically

2. Verify Your Versions

Vulnerable Versions:

3. Patch Immediately

No workarounds exist - upgrading is the only solution:

# For Next.js
npm install next@<patched-version>

# For React (if using RSC directly)
npm install react@19.0.1 react-dom@19.0.1

4. Post-Patch Actions

After patching:

• Rotate all secrets: API keys, database passwords, JWT secrets, etc.

• Review environment variables: Assume all credentials were compromised

• Audit SSH keys: Remove and regenerate all SSH access keys

• Check for persistence: Look for suspicious cron jobs, systemd services, startup scripts

• Monitor processes: Watch for unusual CPU usage or network activity

• Review access logs: Look for suspicious HTTP POST requests to RSC endpoints

5. Deploy Runtime Detection

If using Falco or similar tools, deploy detection rules for React2Shell exploitation attempts.

Signs of Compromise

Check your servers for these indicators:

Process-Level Indicators

# Unusual CPU usage
top
htop

# Suspicious processes
ps aux | grep -E "(xmrig|monero|miner|cpuminer)"

# Network connections to mining pools
netstat -tulpn | grep -E "(moneroocean|pool\.|stratum)"

Persistence Mechanisms

# Check cron jobs
crontab -l
cat /etc/crontab
ls -la /etc/cron.*

# Check systemd services
systemctl list-units --type=service --state=running

# Check startup scripts
cat /etc/rc.local
ls -la /etc/init.d/

File System Artifacts

# Look for recently modified files
find / -mtime -7 -type f 2>/dev/null

# Check for hidden files in common directories
ls -la /tmp
ls -la /var/tmp
ls -la ~/.config

Industry Impact & Statistics

Based on security research:

• 39% of cloud environments contain Next.js or React in vulnerable versions

• 61% of environments with Next.js have publicly accessible instances

• 44% of all cloud environments have publicly exposed Next.js (regardless of version)

• Exploitation observed across AWS, Azure, GCP, and other cloud platforms

• Public RCE exploits available with near-100% reliability

References & Additional Resources

Official Advisories

• Next.js Security Advisory (CVE-2025-66478)

• React Security Advisory (CVE-2025-55182)

• AWS Security Bulletin AWS-2025-030

Security Research

• Wiz Research: React2Shell Analysis

• Microsoft Security Blog: Defending Against React2Shell

• Palo Alto Unit 42: Exploitation Report

Tools

• Official Fix Tool

• Assetnote Scanner

Key Takeaways

1. Patch immediately - This is a maximum severity (10.0) RCE with active exploitation

2. Assume compromise - If you're running vulnerable versions publicly, assume you've been targeted

3. Full rotation - Rotate all secrets, keys, and credentials after patching

4. Consider full rebuild - For critical systems, rebuilding may be safer than remediation

5. Monitor actively - Watch for signs of cryptocurrency mining or unusual activity

6. Stay informed - Follow security advisories for additional related vulnerabilities

Conclusion

The React2Shell vulnerability represents a critical threat to the React and Next.js ecosystem. The ease of exploitation, combined with the widespread adoption of these frameworks, has created an attractive target for threat actors.

Our incident demonstrates that attackers are actively exploiting this vulnerability at scale, targeting cloud infrastructure across multiple providers. The sophistication of the persistence mechanisms we encountered suggests organized threat actors, not just opportunistic attacks.

Do not delay patching. Every hour your vulnerable Next.js application remains exposed is an opportunity for compromise. The attacker we encountered has already compromised 40+ servers - don't let yours be next.

Have you encountered React2Shell exploitation? Share your experience in the comments or reach out to contribute to community awareness.

Update your servers. Rotate your secrets. Stay vigilant.

The emergence of the "oracle problem" has been a central challenge in bridging the gap between the blockchain and the broader digital landscape. Developers have relied on centralized Oracle services to fetch off-chain data and integrate it into their smart contracts, but this approach has introduced new trust assumptions, vulnerabilities, and operational complexities.

The Oracle Problem

This problem arises from the fundamental nature of blockchain technology. Smart contracts, by design, can receive messages and update their internal state, but they cannot directly send messages or interact with external systems that are not native to them. This means that it’s easy for a solidity contract to communicate with another solidity contract, but much more harder and complicated to make an api call to a rest API.

This one-way communication model means that smart contracts are largely isolated from the wealth of data and services available on the open web, which is predominantly hosted on traditional, centralized Web2 platforms. What a limitation right?

Oracles → a solution or a new problem?

To overcome this limitation, developers have turned to Oracle services – centralized entities that act as intermediaries, fetching data from external sources and relaying it to smart contracts. While oracles have enabled some level of integration between blockchains and the broader digital ecosystem, they come with their own set of drawbacks.

Trust Assumptions

Oracles introduce additional trust assumptions, as users must trust the Oracle provider to faithfully retrieve and deliver the requested data. But this goes against the foundational structure of blockchains. Blockchains must aim to be fully trustless.

SPOF

They also create potential single points of failure, as the Oracle service itself can become a vulnerability if compromised or disrupted.

HTTPS Outcalls

The Internet Computer, a revolutionary blockchain platform developed by DFINITY, has introduced a game-changing solution to the oracle problem – HTTPS outcalls. This feature allows smart contracts, known as "canisters" on the Internet Computer, to directly make HTTPS requests to external data sources and services, effectively bridging the gap between the blockchain and the broader Web2 world.

Under the hood, HTTPS outcalls work by placing the HTTP request in the subnet's replicated state. Each replica of the Internet Computer then independently makes the same request and reaches a consensus on the response, which is then delivered back to the originating canister. This approach ensures that the integrity of the data is maintained, as the decentralized network of replicas verifies the response, eliminating the need for a centralized Oracle service.

This approach solves two major problems created by oracles, which are:

1. Trust Assumptions: Now developers no longer need to trust a centralized service to be honest and not manipulate data responses.

2. SPOF: HTTP requests are executed by decentralized replicas on the Internet Computer and need to reach a consensus state before the response is returned. If a replica is down or unreachable, the canister will still work.

Use Cases for HTTPS Outcalls

The ability to make direct HTTPS outcalls from smart contracts opens up a vast array of new use cases for blockchain-powered applications. Some key examples include:

1. Retrieving real-time market data: Canisters can directly fetch cryptocurrency exchange rates, stock prices, and other financial data from Web2 APIs, enabling the development of decentralized finance (DeFi) applications that can accurately track and respond to market fluctuations. An example live on ICP is the Exchange Rate canister - https://internetcomputer.org/docs/current/developer-docs/defi/exchange-rate-canister.

2. Integrating with other blockchains and Web2 services: HTTPS outcalls allow for seamless integration between the Internet Computer and other blockchain networks, as well as traditional Web2 services, enabling cross-chain communication and the development of interoperable dApps.

3. Sending notifications and communications: Canisters can use HTTPS outcalls to trigger external actions, such as sending emails, SMS messages, or push notifications, enhancing the user experience and enabling new use cases for decentralized applications.

4. Accessing IoT data, weather forecasts, sports scores, and more: The possibilities for HTTPS outcalls extend far beyond financial data, as canisters can directly retrieve a wide range of information from various Web2 sources, unlocking new opportunities for innovative dApps.

Success Stories

A. Exchange Rate Canister (XRC)

To demonstrate the power of HTTPS outcalls, the DFINITY team has developed the Exchange Rate Canister (XRC) – a smart contract that fetches real-time cryptocurrency exchange rates directly from major exchanges, without the need for any centralized oracle services.

The XRC canister leverages HTTPS outcalls to query public APIs offered by leading cryptocurrency exchanges, such as Coinbase and Binance, to retrieve the latest prices for a variety of digital assets. It can then provide this data to other canisters, enabling the development of decentralized exchange (DEX) applications and other DeFi tools that require accurate and up-to-date market information.

By eliminating the reliance on oracles, the XRC demonstrates how HTTPS outcalls can simplify the integration of blockchain-based applications with the broader digital ecosystem, paving the way for greater innovation and adoption in the Web3 space.

B. PatriotAI Canister

PatriotAI extensively used HTTPS outcalls to send prompts to OpenAI Assistant API and query run status periodically on a canister. It was able to offer personalized experiences to users of the learning platform by utilizing a generative AI provider, OpenAI, to generate questions to test learners on their progress through course modules.

By eliminating the need for oracles, PatriotAI can provide its users with a trustless and readily available environment to learn and train themselves about corruption.

The Impact of HTTPS Outcalls

The introduction of HTTPS outcalls on the Internet Computer has the potential to revolutionize the way blockchain-based applications interact with the wider digital landscape. By removing the need for centralized oracle services, HTTPS outcalls eliminate the associated trust assumptions, vulnerabilities, and operational complexities, allowing developers to focus on building innovative and reliable decentralized applications.

This paradigm shift unlocks a wealth of new use cases for smart contracts, as they can now directly access a vast array of data and services available on the open web. From DeFi applications that leverage real-time market data to dApps that seamlessly integrate with other blockchains and Web2 services, the possibilities are vast and exciting.

As more developers explore and leverage the power of HTTPS outcalls, we can expect to see a surge of innovation and adoption in the Web3 space. By bridging the gap between blockchain and the broader digital world, HTTPS outcalls have the potential to accelerate the path towards true blockchain singularity, where the majority of computations and interactions occur on decentralized platforms.

Takeaway

The introduction of HTTPS outcalls on the Internet Computer blockchain represents a significant breakthrough in the ongoing quest to integrate blockchain technology with the wider digital ecosystem. By allowing smart contracts to directly fetch data and interact with external services, this feature overcomes the limitations of traditional oracle-based approaches and opens up a world of new possibilities for decentralized applications.

As developers continue to explore and harness the power of HTTPS outcalls, we can expect to see a surge of innovation in the Web3 space, with applications that seamlessly bridge the gap between blockchain and the broader digital landscape. This technology has the potential to catalyze the widespread adoption of decentralized solutions, paving the way for a future where the majority of our digital interactions and computations occur on secure, transparent, and autonomous blockchain networks.

If you're new to the world of crypto and web3, don't worry! We're about to embark on a journey that will demystify these concepts and show you how they're shaping our future. So, grab a cup of your favorite beverage, and let's dive into the exciting world of Neo and its oracle service.

What is Neo?

Before we delve into oracles, let's start with the basics. Neo is a blockchain platform – think of it as a digital ledger that records transactions in a secure, transparent, and unchangeable way. But Neo isn't just any blockchain; it's often called the "Chinese Ethereum" because of its ability to host smart contracts.

Now, smart contracts are like digital vending machines. You put in a coin (or in this case, cryptocurrency), and if certain conditions are met, you get your snack (or the desired outcome of the contract). These contracts can automatically execute actions when specific conditions are fulfilled, without the need for intermediaries.

What makes Neo special is its focus on creating a "Smart Economy." It aims to digitize real-world assets and automate their management through smart contracts. And this is where oracles come into play.

Understanding Oracles

Imagine blockchain as a super-secure, windowless room where all these smart contracts live. They're great at processing information within their own world, but they have no idea what's happening outside. This is where oracles come in – they're like trustworthy messengers that bring real-world information into the blockchain world.

Oracles are crucial because they allow smart contracts to interact with data from the outside world. Without oracles, smart contracts would be limited to working with information already stored on the blockchain. But with oracles, the possibilities become endless!

Neo's Built-In Oracle Service

Now, here's where Neo shines. Many blockchain platforms require developers to use third-party oracle services, which can be complex and costly. But Neo? Neo has a built-in oracle service that comes as part of its core functionality. It's like having a Swiss Army knife when others are carrying a single-blade pocket knife.

Neo's oracle service works by allowing node operators (think of them as the maintainers of the blockchain network) to act as oracle providers. These providers fetch external data and bring it onto the blockchain in a secure and verifiable manner. This data can then be used by smart contracts to execute actions based on real-world events.

Real-World Use Cases

Now, let's explore some exciting ways Neo's oracle service can be used in the real world:

1. Weather-based Insurance: Remember our picnic scenario? With Neo's oracle service, you could create an insurance policy that automatically pays out if it rains on your planned picnic day. The smart contract would use weather data provided by the oracle to determine if it should release the funds.

2. Supply Chain Management: Imagine a smart contract that tracks the temperature of a shipment of vaccines. The oracle could provide real-time temperature data from IoT sensors. If the temperature goes above or below the safe range, the contract could automatically notify the relevant parties or even reroute the shipment.

3. Decentralized Sports Betting: Sports enthusiasts could place bets using smart contracts that automatically pay out based on the final scores of games. The oracle would provide verified sports results, ensuring fair and transparent betting.

4. Real Estate Tokenization: Property values could be updated in real-time based on market data provided by oracles. This would allow for more accurate and dynamic pricing of tokenized real estate assets.

Benefits of Neo's Oracle Service

Neo's built-in oracle service offers several advantages:

• Reliability: Since it's built into the core protocol, it's as reliable as the Neo blockchain itself.

• Cost-effectiveness: Developers don't need to pay for or integrate third-party services, reducing costs and complexity.

• Ease of Use: The service is readily available and easy to implement, lowering the barrier to entry for developers.

Conclusion

As we've seen, Neo's built-in oracle service is a powerful tool that bridges the gap between the digital world of blockchain and our physical reality. It opens up a world of possibilities, from weather-based insurance to supply chain management and beyond.

The future potential of this technology is immense. As more real-world data becomes available to smart contracts, we could see innovations in fields ranging from healthcare to finance, from environmental protection to governance.

So, the next time you hear about blockchain or crypto, remember – it's not just about digital currencies. It's about creating a smarter, more connected world where digital contracts can interact with our physical reality in meaningful ways.

Are you excited about the possibilities? Why not dive deeper into the Neo ecosystem? Explore their documentation, join their community, or even try building your own oracle-powered application. The future is here, and it's more accessible than you might think!

The Power of Neo

Imagine a world where digital assets, smart contracts, and decentralized applications (dApps) coexist seamlessly, powered by a blockchain that's both efficient and developer-friendly. This is the vision that Neo, often called the "Chinese Ethereum," is bringing to life.

But why should you, as a budding developer or curious enthusiast, care about Neo's ecosystem? The answer lies in the growing importance of blockchain technology and the concept of Web3 - the next evolution of the internet. As we move towards a more decentralized digital world, platforms like Neo are paving the way for innovative applications that could reshape industries and empower users in unprecedented ways.

What is Neo? Blockchain Simplified

Before we dive into the developer tools, let's take a moment to understand what Neo is all about. At its core, Neo is a blockchain platform - think of it as a digital ledger that records transactions and data in a secure, transparent, and immutable way. But Neo isn't just any blockchain; it's designed with developers in mind.

What sets Neo apart? Imagine a Swiss Army knife for blockchain development. Neo offers:

1. Support for multiple programming languages (we'll get to that later)

2. A dual-token system (NEO for governance and GAS for transaction fees)

3. A unique consensus mechanism called delegated Byzantine Fault Tolerance (dBFT)

These features make Neo not just powerful, but also accessible to developers from various backgrounds.

Web3 and Neo: A Match Made in Digital Heaven

You've probably heard the term "Web3" buzzing around, but what does it really mean? Think of the internet's evolution:

• Web1: The read-only web (static websites)

• Web2: The read-write web (social media, user-generated content)

• Web3: The read-write-own web (decentralized, user-controlled data and assets)

Web3 is all about giving power back to users, enabling them to own their data, digital assets, and online identities. This is where Neo shines. By providing a platform for building decentralized applications, Neo is helping to bring the Web3 vision to life.

Imagine creating an application where users truly own their in-game items or a social media platform where users control their data. These are the kinds of possibilities that Neo opens up in the Web3 world.

Neo's Developer Toolbox: Your Gateway to Blockchain

Now that we've laid the groundwork, let's explore the tools that make Neo a developer's playground. Think of these as your trusty companions on your blockchain development journey:

Neo-CLI: Your Command Center

Neo-CLI is like the command prompt for Neo. It's a powerful tool that allows developers to interact with the Neo blockchain directly. Whether you're deploying smart contracts, managing wallets, or syncing with the network, Neo-CLI has got you covered.

Neo-GUI: The Friendly Face of Blockchain

For those who prefer a more visual approach, Neo-GUI offers a graphical interface to interact with the Neo blockchain. It's perfect for beginners who want to explore Neo's capabilities without diving into command-line operations.

Neo-Express: Your Blockchain Playground

Imagine having your own private Neo blockchain to experiment with. That's exactly what Neo-Express offers. It's a rapid deployment tool that allows developers to set up a private Neo blockchain instance for testing and development. Think of it as your sandbox where you can build and break things without consequences.

Features

• Blockchain instance management

• Wallet management

• Asset management

• Smart contract management

• Blockchain checkpoint and rollback

Smart Contracts: The Building Blocks of Blockchain Applications

Smart contracts are at the heart of blockchain applications. But what exactly are they? Imagine a vending machine. You put in a coin, press a button, and out comes your snack. Smart contracts work similarly but in the digital realm. They're self-executing contracts with the terms directly written into code.

What makes Neo's approach to smart contracts special is its support for multiple programming languages. While many blockchain platforms require learning a specific language, Neo allows developers to write smart contracts in languages they might already know, such as C#, Python, or Java.

NeoContract, Neo's smart contract system, is designed to be developer-friendly and efficient. It supports features like deterministic finite automata for enhanced performance and a manifest for better security.

To make development even more accessible, Neo offers frameworks tailored to different programming languages:

Neo Blockchain Toolkit: Your All-in-One Development Suite

C#, Python, Go

Imagine having a Swiss Army knife for Neo development - that's exactly what the Neo Blockchain Toolkit (NBT) is. Seamlessly integrated with Visual Studio Code, the most popular code editor, NBT is a one-stop shop for creating and preparing smart contracts for production.

Key features of NBT include:

• Easy deployment of private networks

• Compilation, deployment, and invocation of smart contracts

• Fully integrated debugging experience

• Time-travel debug support

• Source-mapped opcode view

• Automated test harness

Whether you're a seasoned blockchain developer or just starting out, NBT provides a best-in-class development experience that streamlines your workflow and boosts productivity.

NEO•ONE

TypeScript, JavaScript

If TypeScript or JavaScript is your language of choice, NEO•ONE has got you covered. This end-to-end development framework for Neo applications includes:

• Tools for local network setup

• Contract compiling and deploying

• Wallet handling

• Automated testing

• Client APIs to simplify interaction with deployed contracts

NEO•ONE empowers web developers to leverage their existing skills to build blockchain applications, bridging the gap between web and blockchain development.

NeoGo: Go's Gateway to Neo Smart Contracts

Golang

Developed by Neo SPCC, NeoGo is more than just a compiler - it's a fully-featured alternative implementation of the core Neo stack, written in Go. For Go developers looking to step into the world of blockchain, NeoGo offers:

• A highly performant node implementation

• A compiler for Go smart contracts

• An SDK for app integration

NeoGo is well-documented and provides a smooth onramp for Go developers to start building on Neo. Whether you're creating smart contracts or building full-scale decentralized applications, NeoGo equips you with the tools you need to leverage the power of Go in the Neo ecosystem.

Neo-devpack-dotnet: C# Contracts Made Easy

C

For the C# developers out there, the Neo Project offers Neo-devpack-dotnet. This core devpack provides all the tools required to compile smart contracts written in C#. Key features include:

• Production of deployable NEF contract files

• Generation of manifest files

• Emission of debug information in a standard format for use with the Neo Debugger

The familiarity of C# combined with the power of Neo makes this an attractive option for .NET developers venturing into blockchain development.

Boa: Python Slithers onto the Blockchain

Python

Python developers, meet Boa - your bridge to Neo smart contract development. Developed by COZ, Boa is designed to feel as natural as possible for Python programmers. Features include:

• Decorator-assisted contract development

• Helper functions for common blockchain operations

• Type hints for improved code clarity

• Support for testing against the core VM TestEngine

• Generation of standard format debug data for use with the Neo Blockchain Toolkit

With Boa, you can bring Python's simplicity and readability to your blockchain projects, making smart contract development more accessible than ever.

Neow3j Compiler: Java Joins the Blockchain Party

Java, Kotlin

For those more comfortable in the Java world, AxLabs brings you the Neow3j Compiler. As one of the three main components of the Neow3j library, this compiler turns Java contracts into the NEF and manifest files required for deployment on N3 networks. Key features include:

• A Gradle plugin to aid compilation

• Automatic generation of standard debugging information

• Seamless integration with the broader Neow3j ecosystem

The Neow3j Compiler allows Java and Kotlin developers to leverage their existing skills and the robust Java ecosystem in blockchain development.

Neo's Software Development Kits (SDKs): Your Language, Your Choice

One of Neo's greatest strengths is its commitment to language diversity, perfectly exemplified by its range of Software Development Kits (SDKs). These SDKs are like specialized toolboxes, each designed for a specific programming language, enabling developers to interact with the Neo blockchain using their preferred language.

Overview of Neo SDKs

Neo provides SDKs for a variety of popular programming languages. This multilingual approach is part of Neo's strategy to lower the entry barrier for blockchain development. Let's explore the key SDKs in the Neo ecosystem:

Neo RPC SDK: C#'s Bridge to Neo

C

Developed by The Neo Project, the Neo RPC SDK is a dependency library that streamlines the integration of C# applications with Neo. It's well-suited for a variety of projects, including:

• Games

• Wallets

• Other blockchain-integrated applications

Key features of the Neo RPC SDK include:

• Easy construction of transactions

• Simplified invocation of RPC interfaces

• Straightforward calling of deployed contracts

For C# developers looking to bring their applications into the blockchain world, the Neo RPC SDK provides a familiar and powerful toolkit.

Neon-js: JavaScript's Gateway to Neo

JavaScript, TypeScript

For JavaScript developers, COZ offers Neon-js, a lightweight library for Neo blockchain interaction. Designed for both backend and frontend use, Neon-js is versatile and powerful. Its key features include:

• RPC interaction

• Wallet manipulation

• Contract invocation

The NetworkFacade class in Neon-js provides a quick way to bootstrap developers, making it easy to construct interactions with the Neo blockchain. Whether you're building a web application or a Node.js backend, Neon-js equips you with the tools you need.

Mamba: Python's Path to Neo Blockchain

Python

Mamba, also developed by COZ, is the main entry point for Python developers wishing to interact with the Neo blockchain. This comprehensive SDK includes:

• Packages for network connectivity

• Data storage utilities

• Cryptography tools

• Handling of complex data types specific to N3

• Smart contract execution capabilities

• Basic node functionality

Mamba brings the simplicity and readability of Python to Neo blockchain development, making it easier than ever for Python developers to create blockchain-integrated applications.

Neo-gogogo: Go's Toolkit for Neo

Golang

For the Go enthusiasts, Neo Global Development offers Neo-gogogo, a lightweight Go SDK. This comprehensive toolkit provides all the structures and methods required to interface with the Neo blockchain. With Neo-gogogo, you can:

• Manage wallets

• Verify state proofs

• Interact with contracts and tokens

• Build transactions

• Send RPC requests

• And much more

Neo-gogogo makes it easy for Go developers to build any kind of application on the Neo blockchain, from simple scripts to complex decentralized applications.

NeoFS API SDKs: Interfacing with Decentralized Storage

Golang, C

Neo SPCC provides SDKs for interfacing with NeoFS, Neo's decentralized storage system:

1. NeoFS API Go: This SDK implements all the core NeoFS components needed to help applications integrate with NeoFS via its Protobuf-based API. It's perfect for Go developers looking to incorporate decentralized storage into their applications.

2. NeoFS API CSharp: For C# developers, this SDK provides similar functionality, allowing .NET applications to interact seamlessly with NeoFS.

These SDKs open up possibilities for creating applications that leverage both blockchain technology and decentralized storage.

Why SDKs Matter

The importance of these SDKs in the Neo ecosystem cannot be overstated. They serve several crucial functions:

1. Lower Entry Barrier: Developers can start building on Neo using languages they already know.

2. Faster Development: SDKs provide pre-built functions and utilities, saving developers from reinventing the wheel.

3. Best Practices: SDKs often incorporate best practices for interacting with the blockchain, helping developers create more secure and efficient applications.

4. Community Growth: By supporting multiple languages, Neo encourages a diverse developer community, fostering innovation and collaboration.

Choosing Your SDK

When deciding which SDK to use, consider your existing skills and the requirements of your project. Are you building a web application? Neon-js might be your best bet. Working on a data-heavy application in Python? Mamba could be the way to go. Building a Go microservice? Neo-gogogo has got you covered.

Remember, regardless of which SDK you choose, you're tapping into the same powerful Neo blockchain. It's just a matter of which language you're most comfortable speaking.

As you embark on your Neo development journey, take some time to explore these SDKs. Try out a few examples, build some small projects, and see which one feels right for you. After all, the best tool is the one you enjoy using.

The Neo Community: Your Support System

No developer is an island, and in the Neo ecosystem, you're part of a vibrant community. Here's how you can tap into this wealth of knowledge and support:

• Official Documentation: Neo's documentation is comprehensive and regularly updated, serving as your go-to resource for all things Neo.

• Community Forums: Platforms like Neo's subreddit and Discord channel are bustling with developers sharing ideas, solving problems, and discussing the latest in Neo development.

• Social Media: Follow Neo on Twitter and join their Telegram group to stay updated on the latest news and events.

• NeoPod ambassador program

Remember, every expert was once a beginner. Don't hesitate to ask questions and engage with the community - you'll find a supportive group eager to help you grow.

DApps in the Neo Ecosystem: Inspiration for Your Next Project

To give you an idea of what's possible with Neo, let's take a quick look at some notable decentralized applications in the ecosystem:

• NEO Name Service: Think of it as a blockchain-based domain name system, making crypto addresses more human-readable.

• Flamingo Finance: A suite of interoperable protocols for decentralized finance (DeFi) on Neo.

• GhostMarket: An NFT marketplace built on Neo, showcasing the platform's capabilities in handling digital assets.

These dApps demonstrate the versatility of Neo and might spark ideas for your blockchain projects.

Your Neo Adventure Awaits

As we wrap up our exploration of Neo's developer ecosystem, I hope you're feeling inspired and empowered to dive into this exciting world of blockchain development. Neo's commitment to developer-friendliness, combined with its powerful features and vibrant community, makes it an excellent platform for both beginners and experienced developers alike.

Whether you're dreaming of creating the next big DeFi application, a revolutionary NFT marketplace, or something entirely new that the world hasn't seen yet, Neo provides the tools and support to turn your vision into reality.

Remember, the world of blockchain and Web3 is still in its early stages. By starting your journey with Neo today, you're positioning yourself at the forefront of a technological revolution. Who knows? The next groundbreaking dApp could be the one you create.

So, fire up your development environment, join the Neo community, and start building. Your adventure in the world of blockchain development begins now. Welcome to the future of the internet - welcome to Neo.

This article covers how a software engineer can answer this question. If this is your first time hearing about DNS, this is going to be a great ride.

DNS Lookup

Firstly, What is DNS? Domain Name System is a large distributed phonebook that maps domain names to IP addresses. Humans access information online through domain names, such as nytimes.com or espn.com. DNS translates the domain names to IP addresses so browsers can load Internet resources.

When you enter google.com in the browser, let's take Chrome for example. It checks its DNS cache if it exists. If it doesn't it uses the OS DNS service to resolve the domain IP address. If the DNS service can't find it in the cache and host file, then it requests the DNS server configured on the network stack.

TCP/IP

When the browser gets the IP address of the domain name, it uses the IP and port number, which defaults to 80 for HTTP connection and 443 for HTTPS connection, to make a TLS handshake with the web server. This creates a connection session between the browser and the web server for other HTTP requests.

When making this connection the browser using the OS network stack, crafts an IP packet containing the connection message, how the packet will get to the server, and how the server response can get back to the browser.

Firewal

A firewall is a network security software that monitors incoming and outgoing network traffic and decides whether to allow or block specific traffic based on a defined set of security rules.

The network packet sent from the browser needs to pass the server firewall check before it can get to the web server for a connection handshake. If the packet doesn't pass the firewall inspection the traffic is rejected, else approved.

Firewalls typically work on the network layer, the transport layer. However, some are also capable of working as high as the application layer, Layer 7.

HTTPS/SSL

If the connection to be created requires an HTTPS connection, the browser uses port 443 and ensures the packet sent is encrypted. Secure Sockets Layer (SSL) certificates, sometimes called digital certificates, are used to establish an encrypted connection between a browser or user’s computer and a server or website.

You can use the lock icon in Chrome search bar to inspect if the browser established a secure connection with the server. You can also see the certificate authority that provided the Digital certificates to the server.

The encrypted connection helps secure data transmitted over the internet. It helps the server verify that HTTP responses were sent by the real server it is connected to. Also ensure that no other person on the internet can inspect the data transmitted, helping users log in or card information stay secure.

Load-balancer

Let's say the web server is getting too many HTTP requests due to an increasing user base or user activity. One solution is to increase the overall capacity of the server housing the web server, this is known as vertical scaling. But this is not the best solution as it's very expensive and requires heavy migration if it's needed to move the web server data to a new server. A solution most web service providers go for is vertical scaling. Instead of increasing the capability of a single server, they create more servers.

This brings up a new problem, how will the browser locate the web server to connect to? That's where and why a load balancer comes into place. It ensures requests from the internet pass through a single point and then distributes requests to its servers according to its algorithm.

Some load balancers can be configured to direct requests to certain servers depending on the HTTP request. Load balances can be used on different network layers. Layer 7 load balancers route network traffic in a much more sophisticated way than Layer 4 load balancers, particularly applicable to TCP‑based traffic such as HTTP. A Layer 7 load balancer terminates the network traffic and reads the message within.

Web server

Web servers deliver static content, like HTML pages, images, videos, and files. Examples of web servers are Apache and Nginx. It uses the HTTP request to load the static content to send back as an HTTP response. This response is what is later rendered to you in the browser. google.com css, js, and images which are sent back to your browser and are then displayed for you to see and use.

Web servers play a very important role in this whole process, they are the main entity that serves most web content. Static files could have also been configured with CDNs to serve static content faster to you based on your location.

Application server

This is closely related to web servers but serves dynamic content instead. Dynamic content means you might not get the same content as someone living far away from your country or continent. Let's take for example google recognizes a popular holiday in your location, the content you will get will be different from another person in another country that does not celebrate that holiday.

Application servers are also widely used on the internet today to give users personalized experiences when using the internet. It's closely related to the idea of having sessions, where a user is identified by a fixed data like IP address or Authentication data.

Application servers execute programs deployed for them. They send the HTTP requests to these programs which then decide on what HTTP response to send back to the application server. This decision could vary on different data, an example is why Google might use your IP or GPS location to serve you something unique.

Database

A database is an organized collection of structured information, or data, typically stored electronically in a computer system. Programs executed by the application server might require a connection to a database to get information related to HTTP requests. This allows for persistent dynamic content.

From the example used earlier, the information regarding the holidays of several countries and the content to be displayed stay in a database, making it persistent and easily accessible by the program.

Conclusion

In conclusion, understanding the intricate workings of the internet's infrastructure, from DNS lookups to load balancing, and the critical components that power it, such as web servers, application servers, and databases, is paramount for any software engineer. This knowledge not only showcases a deep understanding of the technological ecosystem but also demonstrates a level of expertise that can earn respect and trust from employers.

DNS, the backbone of the internet, translates human-readable domain names into IP addresses, allowing us to access online resources effortlessly. The journey from a user's query to the actual web server involves several crucial steps, each with its significance.

TCP/IP and firewalls play essential roles in establishing secure connections and ensuring data integrity. The use of HTTPS/SSL certificates further enhances security and privacy for users, safeguarding their data from potential threats.

In the face of increasing user demands, load balancers come to the rescue by efficiently distributing traffic across multiple servers, ensuring a smooth and responsive user experience. This scaling strategy is not only cost-effective but also minimizes the need for complex server migrations.

Web servers deliver static content, while application servers provide dynamic, personalized experiences for users. These elements, along with databases, form the foundation of modern web applications, enabling the persistent storage and retrieval of information.

In a world where digital interactions have become a fundamental part of our lives, comprehending the intricate web of technology that underlies it all is not only a valuable skill but a testament to a software engineer's prowess in shaping the digital landscape. With this knowledge, engineers can create robust, secure, and efficient systems that drive the Internet forward, making it a safer and more dynamic environment for all its users.

CI/CD is automating what developers already do. For example, running regression tests, and deploying new features. It’s important because it saves time and eliminates the possibility of human errors. Have you ever experienced forgetting to add an env key when deploying that new feature that depends on a new secret key? These kinds of little mistakes might cost users' trust.

You can also use CI/CD to automate your code testing. No need to run tests manually for every single commit to the repository. You can run automation for tests using tools like GitHub Actions, and Jenkins.

This article covers how you can use GitHub action to automate tests and builds in your C projects.

You can use GitHub Actions to automate many development workflows. This article covers how to use it with C programming language by explaining the basics of GitHub Actions. If you understand the basics you will be able to use it for other projects written in different programming languages or frameworks.

Prerequisites 💻

To follow this article, you need to have;

1. an understanding of writing C,

2. an understanding of how to build C programs with GCC,

3. gcc installed on your machine, and

4. a little understanding of Git and GitHub.

Project Setup 🤓

You need to have a codebase with tests first before automating its testing. There is a simple C codebase already provided. You can clone it to get started, but to work with GitHub Actions you need to fork the repository. This is because you need permission to trigger workflow runs on a particular repository. By forking the repository you have access to trigger workflow runs on the forked repository. This is needed so you can visualize and test your workflow runs. The next section covers what is called a workflow.

Repository to fork: https://github.com/devvspaces/dsa-with-c

Description: This is a repository that implements many data structures and algorithms in C and includes tests.

This repository already has GitHub Actions workflow configured for testing. You need to delete the .github folder in the root directory to start afresh.

What Is GitHub Action? 🤔

GitHub Actions helps you to automate your development workflows.

GitHub Actions makes it easy to automate all your software workflows, now with world-class CI/CD. Build, test, and deploy your code right from GitHub. Make code reviews, branch management, and issue triaging work the way you want. - GitHub Actions documentation

GitHub actions can be used to automate many software workflows. A single automation flow from start to end is called a workflow run in GitHub Actions. Developers can configure multiple workflows, each for separate tasks like regression testing and deployment. Each workflow can have one or more jobs.

Workflows can be configured to be triggered by an event in your GitHub repository. For example, you can have one workflow to build and test when there is a new pull request. Then another workflow to deploy your application every time you push a new commit. Also, another workflow to add a label every time someone opens a new issue.

Events are specific operations that trigger a workflow run. Events can be a new pull request or a new build release. You can configure workflows to listen to only events originating from specific branches or tags.

Jobs in a workflow are a group of steps executed on the same runner. A step can be a script or an action. For example, there can be a step for building a C source code and another step for executing the build.

Actions are a complex set of steps that are often repeated. For example, checking out a repository on a runner. Actions are reusable in many workflows and repositories. A developer does not need to rewrite the steps for checking out a repository on a runner, there are already efficient actions for this.

Runners are servers that execute jobs in a workflow when it’s triggered. A runner can only run a single job at a time. GitHub offers Ubuntu Linux, Microsoft Windows, and macOS runners to run your workflows. Each workflow run executes in a fresh, newly-provisioned virtual machine.

Setting Up Your Workflow 💻

To create a new workflow in your repository you need to create a .github/workflows directory in your root directory. Then create a new file called ctest.yml. The file name is at your discretion but it must have a .yml extension. This file will contain configurations to build and test the source code in the repository.

name specifies the name of a workflow file.

# ctest.yml
name: C Build Test

The on key configures events and resources that trigger workflow runs.

The code below configures a workflow run when there is a push or pull request to the master branch.

# ctest.yml
...
on:
  push:
    branches: [ "master" ]
  pull_request:
    branches: [ "master" ]

To specify each job and its respective steps, you can use the job's key and steps key for each job. This is where most configurations exist. Steps like checking out your repository on a runner and executing scripts or commands are here.

jobs groups together all the jobs that run in a workflow.

build-and-execute defines this job's name. The child keys define the properties of this job. This job handles building and executing the code because it’s faster and each job runs on its runner. Meaning, jobs can’t access each other’s files.

runs-on configures the job to run on the latest version of an Ubuntu Linux runner.

steps groups together all the steps that run in the build-and-execute job. Each item nested under this section is a separate action or shell script.

uses specifies this step to run v3 of the actions/checkout action. This action checks out your repository onto the runner, allowing you to run scripts or other actions against your code (such as build and test tools). You should use the checkout action any time your workflow will use the repository's code.

The run keyword instructs this step to execute a command on the runner. In this case, gcc compiles and build all the c files in the project to an executable called dsa. Then the next step executes this executable. The last step deletes the build from the runner, but this step is not required. The runner deletes all the files created in the runner when each job is complete.

# ctest.yml
...
jobs:
  build-and-execute:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - name: Build source code
      run: gcc -Wall -Werror -Wextra -pedantic -std=gnu89 */*.c */*/*.c */*/*/*.c -o dsa
    - name: Run build
      run: ./dsa
    - name: Clean build
      run: rm -f dsa

Your complete workflow file should now look like this.

# ctest.yml
name: C Build Test

on:
  push:
    branches: [ "master" ]
  pull_request:
    branches: [ "master" ]

jobs:
  build-and-execute:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - name: Build source code
      run: gcc -Wall -Werror -Wextra -pedantic -std=gnu89 */*.c */*/*.c */*/*/*.c -o dsa
    - name: Run build
      run: ./dsa
    - name: Clean build
      run: rm -f dsa

Now you can commit and push.

Visualizing Your Workflow Run 📈

To visualize your new workflow run. Head over to your repository on GitHub. Under your repository name, click the actions tab. There you will see the new workflow run triggered by the push event you initiated.

From the list of workflow runs, click the name of the run to see the workflow run summary. In the left sidebar, you will the list of jobs in execution for the workflow. Click the job you want to see. In this case that will be the build-and-execute job. To view the results of a step, click the step. All standard output you will get on a normal machine is available under each step in the job.

Conclusion 👋

Hope you now understand how GitHub Actions work and how to configure a simple workflow. It’s easy to use GitHub Actions to automate workflows for many programming languages and frameworks.

You can always check the documentation for more understanding of writing complex workflows. It’s recommended to check out popular open-source frameworks to learn more about using GitHub Actions.

Resources

GitHub Actions Documentation

Microsoft Github Actions Course

Using Github Actions with Django

The amount of data a processor can execute at once depends on its number of virtual memory addresses. The maximum integer representation a 32-bit processor can work with is 4,294,967,295 (2^32 − 1), while for a 64-bit processor is 18,446,744,073,709,551,615 (2^64 − 1). These bit constraints affect C standard integer and float data types.

Most high-level languages, like Python, allows programmers to work with larger numbers. Allowing programmers to write programs for processing numbers larger than what the CPU can handle. It's possible to write programs for calculating big numbers in C, but it can get very complex and confusing.

GMP, GNU Multiple Precision Arithmetic Library, streamlines the complexity of working with big numbers in C. This article covers how you can use the GMP Library to perform basic mathematical operations on various kinds of numbers.

Prerequisites

To follow this article, you need to have;

• A basic understanding of C,

• GCC installed - for compiling C programs,

• Basic understanding of GCC,

• Access to the internet - for installing GMP, and

• A text editor was installed.

What is GMP?

GMP is a free library for arbitrary precision arithmetic, operating on signed integers, rational numbers, and floating-point numbers. There is no practical limit to the precision except the ones implied by the available memory in the machine GMP runs on. GMP has a rich set of functions, and the functions have a regular interface. - gmplib.org

GMP allows you to calculate big numbers of many kinds with high precision. It represents numbers in its program as strings.

Installing GMP

To install the GMP library, you can follow this informative guide. It explains how you can download, configure and build the library on Windows, Mac, and Linux.

Performing Arithmetical Operations Using GMP

After installing the library, you need to include the GMP header file in your program to use the library for calculations.

Addition

For adding two integers, you need to declare and initialize the variables in a special way before calculations.

Include GMP library header. This allows you to access the types and macros declared in the header file.

#include <gmp.h>

Declare and initialize the variables for the result, and the numbers to add together.

mpz_t res, num_a, num_b;

/* Initializing integers */
mpz_init(res), mpz_init(num_a), mpz_init(num_b);

Initializing the variables using mpz_init function tells GMP to allocate memory space to the variables. Then you can assign any large number to the initialized variables.

The lines below assign 3479349783486297692763769376273723897236 to num_a and 9223372036854775807 to num_b. There is no limit to the amount of precision you get when using GMP for calculations.

/* Assigning integers */
mpz_set_str(num_a, "3479349783486297692763769376273723897236", 10);
mpz_set_str(num_b, "9223372036854775807", 10);

The mpz_set_str function takes the first argument as the variable to assign a value. The second argument is the string representation of the integer you want to assign. The third argument represents the base of the integer (which, in this case, is the second argument). In the above example, the integers are in base 10, meaning decimal numbers.

Now you can add the number together using the mpz_add function from the GMP library. The function sets res to num_a + num_b. Then you can use gmp_printf to print the values. It works like C standard library printf function. The only difference is that gmp_printf allows you to format larger numbers. The %Zd flag in the format string in gmp_printf is to format large integers.

/* Addition */
mpz_add(res, num_a, num_b);
gmp_printf("Addition:\n%Zd + %Zd = %Zd\n\n", num_a, num_b, res);

Now you need to clear the variables. Clearing the variables frees the memory allocated for the variables.

/* Clearing memory */
mpz_clear(res), mpz_clear(num_a), mpz_clear(num_b);

The full program should look like this.

// main.c

#include <gmp.h>

int main(void)
{
    mpz_t res, num_a, num_b;

    /* Initializing integers */
    mpz_init(res), mpz_init(num_a), mpz_init(num_b);

    /* Assigning integers */
    mpz_set_str(num_a, "3479349783486297692763769376273723897236", 10);
    mpz_set_str(num_b, "9223372036854775807", 10);

    /* Addition */
    mpz_add(res, num_a, num_b);
    gmp_printf("Addition:\n%Zd + %Zd = %Zd\n\n", num_a, num_b, res);

    /* Clearing memory */
    mpz_clear(res), mpz_clear(num_a), mpz_clear(num_b);

    return (0);
}

You need to link the GMP library when compiling your program. You can only link the dynamic or static library if you have installed GMP on your machine. If you haven’t, go back to the installation section above.

Run the command below to build your program. The -lgmp flag tells gcc to search for the GMP library when linking. GCC will search the list of standard directories to find the library.

gcc main.c -o main -lgmp

If it could not find the GMP library on your machine. Try with the build flags below. The -L~/gmp-6.2.1/.libs flag tells gcc to also check the ~/gmp-6.2.1/.libs directory if it could not find the library using the standard list. For you, this path is the location you installed GMP.

gcc main.c -o main -L~/gmp-6.2.1/.libs -lgmp -static

That covers how you can add large numbers in C and build your program.

Subtraction

You can use the mpz_sub function from the GMP library to subtract large numbers. Ensure you initialize the variables and assign their integers respectively.

This subtracts num_b from num_a and assigns the result to res.

/* Subtraction */
mpz_sub(res, num_b, num_a);
gmp_printf("Substraction:\n%Zd - %Zd = %Zd\n\n", num_a, num_b, res);

Your full program might look like this.

// main.c

#include <gmp.h>

int main(void)
{
    mpz_t res, num_a, num_b;

    /* Initializing integers */
    mpz_init(res), mpz_init(num_a), mpz_init(num_b);

    /* Assigning integers */
    mpz_set_str(num_a, "3434234245245245252452452452", 10);
    mpz_set_str(num_b, "9999943444444434344444444434232323232322", 10);

    /* Subtraction */
    mpz_sub(res, num_a, num_b);
    gmp_printf("Substraction:\n%Zd - %Zd = %Zd\n\n", num_a, num_b, res);

    /* Clearing memory */
    mpz_clear(res), mpz_clear(num_a), mpz_clear(num_b);

    return (0);
}

Multiplication

For multiplying integers, use mpz_mul function from the GMP library.

This multiplies num_a and num_b and assigns results to res.

/* Multiplication */
mpz_mul(res, num_a, num_b);
gmp_printf("Multiplication:\n%Zd x %Zd = %Zd\n\n", num_a, num_b, res);

Division

For division, things will look a little different. Instead of using the type mpz_t to declare integers as you have seen before, you will use mpf_t instead to declare floats.

To divide large numbers using GMP you need to declare and initialize floats and not integers.

mpf_t div_res, div_a, div_b;

Then you initialize the variables using mpf_init.

/* Initializing floats */
mpf_init(div_res), mpf_init(div_a), mpf_init(div_b);

Assign values you want to the variables. They can be as large as needed. The last argument to mpf_set_str tells GMP that the string is in a decimal format.

/* Assign floats */
mpf_set_str(div_a, "7347973477376467734.34322335550000000000", 10);
mpf_set_str(div_b, "489384838948848.4783873847", 10);

Divide and print out the results using the functions below. The GMP function mpf_div divides div_a by div_b and assigns the result to div_res. Then prints out the formatted string.

/* Division */
mpf_div(div_res, div_a, div_b);
gmp_printf("Division:\n%Ff / %Ff = %Ff\n\n", div_a, div_b, div_res);

Your final code should look like this.

// main.c

#include <gmp.h>

int main(void)
{
    mpf_t div_res, div_a, div_b;

    /* Initializing floats */
    mpf_init(div_res), mpf_init(div_a), mpf_init(div_b);

    /* Assign floats */
    mpf_set_str(div_a, "7347973477376467734.34322335550000000000", 10);
    mpf_set_str(div_b, "489384838948848.4783873847", 10);

    /* Division */
    mpf_div(div_res, div_a, div_b);
    gmp_printf("Division:\n%Ff / %Ff = %Ff\n\n", div_a, div_b, div_res);

    /* Clearing memory */
    mpf_clear(div_res), mpf_clear(div_a), mpf_clear(div_b);

    return (0);
}

Square root

Ensure you declare and initialize the float variables. Then calculate the square root of that variable using mpf_sqrt function. This function only accepts two arguments. The variable to square root and the variable to assign the result.

/* Squre root */
mpf_sqrt(div_res, div_a);
gmp_printf("Square root:\nsqrt(%Ff) = %Ff\n", div_a, div_res);

Conclusion

You can perform other arithmetical operations that are not covered in this article using GMP. Check out the resources section below for their documentation to learn more. It covers how you can perform more Arithmetical operations with Logical and Bitwise operations. Also, you can calculate using C int types and rational numbers.

Use the resources provided below to learn more about working with big numbers in C using the GMP library. If you are looking for a challenge try writing a C function to add two large numbers.

Thanks for reading this article. 😊

Resources

• Linking libraries using GCC Compiler - https://gcc.gnu.org/onlinedocs/gcc/Link-Options.html

• Installing GMP from the official website - https://gmplib.org/#DOWNLOAD

• Guide to installing GMP on any machine - http://rstudio-pubs-static.s3.amazonaws.com/493124_a46782f9253a4b8193595b6b2a037d58.html

Testing and Deployment are part of the cycle discussed earlier. We build, deploy and test our applications a lot. Doing this every time is vulnerable to human error. Automating our tests guarantees that every commit undergoes testing.

This article covers how we can use GitHub Actions to automatically test our Django applications.

Let’s dive in.

Prerequisites

To follow through with this article, we need the following:

• Basic knowledge of Django, Git, and GitHub

• A GitHub account

• Python3 and Django installed on our operating system

Setup Demo Django Application

We need a Django application that has tests to understand how GitHub actions work. We can work with this demo Django project, which includes tests.

GitHub actions run on GitHub repositories and not on our machines. Only the owner of the repository can manage workflow runs. Later in the article, we will cover what Workflows are.

We need to fork the repository and clone it.

git clone <REPOSITORY_URL>

We should set up a new virtual environment.

# linux and mac
cd <PROJECT_DIR>
python3 -m venv venv
source venv/bin/activate

# windows
cd <PROJECT_DIR>
python3 -m venv venv
venv/Scripts/activate  # <-- difference

Now we need to install the project dependencies. Run this command in the project directory. This will install all the dependencies specified in requirements.txt file.

pip install -r requirements.txt

Now that our application setup is complete. Let’s run our tests will. The tests confirm the reliability of our application.

cd project
python manage.py test

The tests are successful. We can now go ahead and automate this process using GitHub actions.😊

Automated tests don’t only work when we use Django’s built-in test runners. We can write automation for any kind of test runners we are using. One popular test runner in the Python community is Pytest.

What are GitHub Actions?

To get started, we need to define a workflow. A workflow is a configurable automated process that will run one or more jobs. A step is a command or action that performs a basic task. An Action is a collection of steps, which makes an action reusable. A Job is an operation in a workflow that contains steps. Jobs can run in parallel or in sequence.

We define workflows by using a YAML file checked in the repository. A workflow will execute when triggered by an event in the same repository it exists.

GitHub workflows listen to many events, examples are push and pull requests. The events configured into our YAML file triggers our workflow. For example, when we push a new commit to the master branch, this can trigger a testing workflow.

Creating A Testing Workflow

Let’s create a workflow that executes our Django application tests.

Workflows are not for testing but for executing jobs which can be anything. An example of this is running tests. Our tests are already defined. We configure workflows to execute them. Other use cases are deployments and closing pull requests.

Create a new directory called .github/workflows in the root directory. Create a new file called test.yml in the .github/workflows directory. This file will be our workflow YAML configuration.

# test.yml
name: Run Django Tests
on:
  push:
    branches: [master]
jobs:
  test-django-app:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.9"]
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v3
        with:
          python-version: ${{ matrix.python-version }}
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
      - name: Run Django test
        run: |
          cd project
          python manage.py test

Let’s break down what each syntax means and how this executes the application tests.

We start by defining the workflow name by using the name syntax.

name: Run Django Tests

The workflow name helps us identify the workflow in our repository’s GitHub Actions tab.

The GitHub Actions tab helps us to manage workflow runs. See how to manage a workflow run on GitHub.

Next, we define the event that triggers the workflow.

on:
  push:
    branches: [master]

By using the on syntax, every commit we push to the master branch triggers the workflow run. We can specify many events and branches.

Next, we have the jobs syntax.

jobs:
  test-django-app:

A scenario to use this is when we want to deploy a web application after successful tests. We can create two jobs, one for testing and the other for deployment. Then configure the deployment to run after when tests are complete and successful.

The name of our job is test-django-app, we can name it anything. But make sure it’s descriptive, as it helps us identify the currently running job in the GitHub Actions tab.

Next, the runs-on syntax defines what machine will run the job. Here we instructed it to use ubuntu-latest version.

runs-on: ubuntu-latest

Next, the strategy syntax helps us define what Python versions we will use to test. In our workflow, we have defined only one version. The job will use each Python version declared in the list to execute itself in parallel.

strategy:
  fail-fast: false
  matrix:
    python-version: ["3.9"]

fail-fast applies to the entire matrix. If fail-fast is set to true, GitHub will cancel all in-progress and queued jobs in the matrix if any job in the matrix fails. This property defaults to true.

Assuming that we built a Python package and set many Python versions. We can ensure the package undergoes testing across defined Python versions.

Jobs contain a sequence of tasks called steps. Steps can run commands, set up docker, or run an action in our repository. These steps run one after the other based on their position in the YAML definition.

The following are the actions taken by each step in our workflow:

1. Checkout the repository on the machine defined for the job,

2. Setup Python using the current version from the strategy,

3. Install pip and the project dependencies,

4. Move into the project directory and run the tests.

We need to commit and push a new workflow to trigger our workflow.

git add .
git commit -m "Added tests automation using Github Actions"
git push

We should see a new running or completed workflow run on the GitHub actions tab in our forked repository.

This workflow would show us each job and the steps executed.

![GitHub workflow job management tab(https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5tlk6am6n0dhwjn7s2mn.png)

Now, any time we push a new commit to the master branch. Our workflow executes our Django application tests.

Understanding GitHub Actions is a complete guide to learning actions, workflows, and jobs. Check it out to master project development automation.

Conclusion

In this article, we have covered how GitHub actions execute Django application tests. Continuous Integration and Delivery is an important part of a modern development cycle. Automating development life cycles ensures we build reliable applications when working with collaborators.

Resources

• Understanding GitHub Actions by GitHub

• Introduction to GitHub Actions Training by Microsoft

• GitHub Actions Tutorial for Beginners to Automate Your Workflow

One major challenge developers face is conflicting environment dependencies, which can cause project failure when deployed. Although Docker solves this problem, its complex setup requirements make it inaccessible to many developers.

This tutorial offers a solution by demonstrating how to dynamically install Python dependencies based on the environment in which the project will run.

Prerequisites

With only a basic understanding of Python and Git, any developer can follow the methods outlined in this tutorial to improve their project's deployment success. The only requirement to have installed for this tutorial is Python. To install Python, visit their official downloads page.

What is the problem?

When using Postgres database with Django, psycopg2 database adapter is a required dependency for this to work. But there is a problem, psycopg2 will install successfully on a Windows environment but won't on a Linux environment, you will need to install psycopg2-binary instead.

Well, if your development and deployment environment is similar or you use Docker, you won't have a problem with this situation. The problem occurs when you have differing environments or multiple collaborators working on the project with you. You don't want them editing the requirements.txt file so as not to cause git conflicts.

How can this problem be solved?

Two major methods will be shared today to help you dynamically install Python dependencies based on your environment or Python version.

Conditional pip requirements syntax

Suppose that the dependencies listed in your requirements.txt file are as follows:

requests==2.26.0
six==1.16.0
urllib3==1.26.7
psycopg2==2.9.6

That can be modified to ensure that when you install the requirements on a Linux environment, the required database adapter for Linux will be installed.

requests==2.26.0
six==1.16.0
urllib3==1.26.7
psycopg2-binary==2.9.3; sys_platform == "linux"
psycopg2==2.9.6; sys_platform == "win32"

You can also program the dependencies to be installed based on the Python version installed on the environment.

requests==2.26.0; python_version > '3.0'
requests==2.26.0; python_version == '2.6'
six==1.16.0
urllib3==1.26.7
psycopg2-binary==2.9.3; sys_platform == "linux"
psycopg2==2.9.6; sys_platform == "win32"

They are defined in PEP 508 and PEP 0345 (Environment Markers), but the syntax appears to follow the draft PEP 0496.

To know all the possible values for sys_platform``, visit this StackOverflow answer.

Now when you run pip install -r requirements.txt, depending on your environment, the appropriate dependency and version will be installed.

This is the easiest and recommended method for dynamically installing dependencies based on the environment or Python version.

Python install script

You could create an install.py script to install dependencies based on the environment or Python version.

• Create a file install.py in your project's root directory

  ```python import pip

  all = ( "requests==2.26.0", "six==1.16.0", "urllib3==1.26.7", ) # All platforms

  windows = ("psycopg2==2.9.6",) # Windows

  linux = ("psycopg2-binary==2.9.3",) # Linux

  darwin = [] # MacOS

def install(packages): print("Installing packages: {}".format(packages)) for package in packages: pip.main(['install', package])

if name == 'main':

from sys import platform

print("Platform: {}".format(platform))

install(all)

if platform == 'win32': install(windows) if platform.startswith('linux'): install(linux) if platform == 'darwin': install(darwin)

print("Completed installation of packages.")

This script works by checking what platform the program is running on using the Python built-in sys package. So it installs the defined dependencies for that platform by calling `pip` by script. This method is more complex and time-consuming, but it gives you more control and flexibility on the program control flow.

* Run the program to install the packages

    ```bash
    python install.py

Conclusion

In conclusion, conflicting environment dependencies can be a huge headache for software developers when deploying their projects. Fortunately, several solutions are available, including Docker and the dynamic installation of dependencies based on the environment. In this article, we have provided two different methods for dynamically installing Python dependencies based on the environment, which can help avoid conflicts and ensure the project runs smoothly across different platforms. These methods allow developers to save time, reduce errors, and focus on building great software. I hope that this article has been helpful in providing useful tips and tricks for managing environment dependencies in your Python projects.

Thanks for reading, I'd also love to connect with you at Twitter | LinkedIn | GitHub | Portfolio. 🤓

What is MindsDB?

MindsDB is an open-source, automated machine-learning platform that enables developers to build and integrate machine-learning models into their applications quickly and easily. MindsDB provides a simple interface that allows you to train, test, and deploy machine learning models without having to write any code. It also supports a wide range of use cases, such as classification, regression, and time series forecasting.

What's the Problem?

New programmers end up in something called the Tutorial Hell. Whereby they keep watching and reading various online tutorials from various content creators. Tutorials are good but most don't cover all the aspects a new programmer needs to suceed in a skill.

What's the Solution?

This project is built with the goal of helping new programmers find quality tech curriculums and resources. The platform curates the best resources and organizes them into curriculums based on user feedback. To improve the user experience and accelerate app growth, the platform integrates MindsDB models for sentiment and text classification on user reviews.

Why do we need to classify Reviews?

When users are progressing through a curriculum or have completed it. Their reviews are important so that we can use them to reorganize the curriculums based on user needs and their learning experience.

Using AI to classify user reviews sentimental can help us categorize or filter reviews by priority and what we need to work on ASAP. Doing this manually would be time-consuming and slow down problem analysis and updates.

Why MindsDB Models?

In this project, we used MindsDB models for sentiment and text classification to classify user curriculum reviews. The user reviews are classified by labels and sentiment. MindsDB models are used because it's easier to integrate than to install Pytorch and LLM (Large Language models).

We can train our models right in the MindsDB SQL editor. Yes, we will use SQL queries to train our AI models. MindsDB NLP engine is currently powered by Hugging Face and Open AI. It supports NLP tasks like Text Classification and Zero-Shot Classification.

By using MindsDB Python SDK, we can easily connect to our MindsDB Cloud Server and execute AI queries on our pre-trained models with just a few lines of Python code. MindsDB's large community and learning resources can help us to find solutions on how to use them to solve any problems we might encounter.

Building The Project

We will cover the project build-up from the DB design to the product mvp. The repository for the completed project Backend API and Frontend Web contain installation guides to set up the project and start using it.

Project Prerequisites

To follow through with this article, we need:

• A MindsDB cloud account. Create a free cloud account at cloud.mindsdb.com.

• Python and NodeJs installed.

• Redis for caching

• Basic knowledge of Django, Django Rest Framework, React, and NextJs frameworks

Django Memcache can be configured instead of redis if problem occurs when installing redis.

Let's start building 🤓💻

Productivity Foundations

Database Design

First of we need to have an understanding of how the Database will be structured. By using dbdiagram.io, we can design relation database structures. Here is a database diagram that we can use.

This can help us get clarity on how the project will be structured. That's like 50% of the problem-solving done here. We would find out tables related to each other and how to categorize the apps in Django. This saved will save a lot of time instead of jumping on the project without direction.

Need the DB Schema used to generate the DB diagram - visit here.

Flow Design

Here is the link to the Figma jam.

Having a simple design for the application flow will go a long way to help us have a working application in our minds. Also finding solutions to other basic problems like the best authentication method to use, how to organize the API, and how the user will use the application.

Rest API Development

Django and Django Rest Framework are used for API development and documentation. Swagger allows us to test the API endpoints from the documentation itself. The backend repository has an installation and usage guide in README.md for setting up the project and testing it with ease.

The API is classified into three major sections:

• Account - Authentication and Authorization

• Curriculum - Organizing curriculums, enrollments, quizzes, and reviews.

• MindsDB Classify - Classifying texts by sentiments and labels.

Account

When a user registers, they will get an OTP in their email to verify their email. The user's OTP is cached for future validations. After validation, the OTP is deleted from the cache. This Django app has the following features:

1. Register

2. Login

3. Forget password

4. Change password

5. Updating user profile

We used JWT (JSON Web Tokens) for user authentication. This allows us to have a stateless authentication that the frontend can easily manage. Simple JWT allows us to use JWT in Django Rest Framework.

Curriculums

This is the main application in the Django project. This application supports the following features:

• user enrollments, progress tracking,

• quiz creation, quiz submission,

• curriculum, syllabus, resources creation,

• reviews submission and classification.

The backend repository contains comments that explain how the Models, Views, and Serializers work.

Basically, the user can search for curriculums and enroll in them. An enrolled user can progress through the curriculum syllabus by completing quizzes for each lesson. At the end or anytime while taking the curriculum they can submit reviews any kind of reviews. When a new review is submitted it is classified and updated in the database.

NLP Classification using MindsDB

Our reviews are classified and stored in the CurriculumReview table.

class CurriculumReview(models.Model):
    """Curriculum Review Model"""

    SENTIMENT = (
        ('POS', 'Positive'),
        ('NEG', 'Negative'),
        ('NEU', 'Neutral'),
    )

    LABEL = (
        ('A', 'Course Content'),
        ('B', 'Exercises'),
        ('C', 'Course Structure'),
        ('D', 'Learning Experience'),
        ('E', 'Support'),
    )

    enrollment = models.ForeignKey(
        CurriculumEnrollment, on_delete=models.CASCADE)
    rating = models.PositiveSmallIntegerField(
        default=0,
        validators=[MinValueValidator(1), MaxValueValidator(5)]
    )
    review = models.TextField(null=False, blank=False)
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)
    sentiment = models.CharField(
        max_length=3, null=True, blank=True,
        choices=SENTIMENT
    )
    label = models.CharField(
        max_length=1, choices=LABEL, null=True, blank=True)

For the sentiment labels, we have:

SENTIMENT = (
    ('POS', 'Positive'),
    ('NEG', 'Negative'),
    ('NEU', 'Neutral'),
)

For the Zero-Shot classification labels, we have:

LABEL = (
    ('A', 'Course Content'),
    ('B', 'Exercises'),
    ('C', 'Course Structure'),
    ('D', 'Learning Experience'),
    ('E', 'Support'),
)

These can then be used as Django Charfield choices, which we can configure in the Django admin for filtering. Allowing us to filter reviews by their labels.

Also, we will see how can reuse this when querying our MindsDB server.

Creating and Training the MindsDB AI Models

First of all, the MindsDB documentation is packed with resources to understand how we can use it.

Today we will be using the following NLP models from Hugging Face:

• Sentiment Classification - cardiffnlp/twitter-roberta-base-sentiment

• Zero-Shot Classification - facebook/bart-large-mnli

As we have read earlier, MindsDB works seamlessly with Hugging Face NLP models. By running SQL queries in our MindsDB Cloud SQL editor we can easily create and train these models to be used in our project.

1. To access our Cloud editor we need to sign in - https://cloud.mindsdb.com/login. Don't have an account yet? Create one here https://cloud.mindsdb.com.

2. After signing in, we can see our free demo instance is already set up and running. Click it to access the SQL Editor.

3. 

   On the right panel in the SQL Editor, we can find the Learning Hub with tutorials and resources for using MindsDB SQL Editor.

4. 

   We can create files, databases, and projects in the SQL Editor. By default, we have a project, mindsdb, already created for us. A project is a collection of data that includes all the necessary information for building, training, and testing a machine-learning model.

A. Training the Sentiment model

Copy and paste the SQL statement below into the editor.

CREATE MODEL mindsdb.hf_sentiment
PREDICT PRED
USING
engine = 'huggingface',
task = 'text-classification',
model_name = 'cardiffnlp/twitter-roberta-base-sentiment',
input_column = 'text',
labels = ['Negative', 'Neutral', 'Positive'];

The labels are from the labels we want to use for sentiment in our Django project.

In mindsdb.hf_sentiment - mindsdb is the project name while hf_sentiment is the model name. We can use any name for the model name, but we need to take note of it because it will be required in the Django project to access the model.

To run the SQL statement, we need to highlight it in the SQL editor and click on Run towards the top of the screen or Shift + Enter.

Running the statement creates a new model. To check the status of the hf_sentiment model, run the following statement.

SELECT *
FROM mindsdb.models 
WHERE name = 'hf_sentiment';

Status = complete tells us that the model training is complete.

To test the model, we run the statement below

SELECT *
FROM mindsdb.hf_sentiment
WHERE text = 'I like you. I love you.';

This results in a response that contains the prediction PRED_explain.

The prediction tells us that the text sounds more Positive than Neutral or Negative. In our Django project, we will need to write a program that retrieves the label with the maximum result.

B. Training the Zero-Shot classification model

Using what we have learned when creating the sentiment model. Let's create one for Zero-Shot too using a similar process.

Create -> Check Status -> Test

To create the model, run the following statement.

CREATE MODEL mindsdb.hf_classify
PREDICT PRED
USING
engine = 'huggingface',
task = 'zero-shot-classification',
model_name = 'facebook/bart-large-mnli',
input_column = 'text',
candidate_labels = ['Course Content', 'Exercises', 'Course Structure', 'Learning Experience', 'Support'];

We also used our Django project labels as the hf_classify labels here too. Also don't forget to note the model name hf_classify.

To check for the status of the newly created model, run the following statement.

SELECT *
FROM mindsdb.models 
WHERE name = 'hf_classify';

Wait until it's completed training.

To test the model, run the following statement.

SELECT *
FROM mindsdb.hf_classify
WHERE text = 'This is a nice curriculum. Explains a lot about HTML tags and css design. But needs more video contents. Thanks.';

We should get the following response.

Telling us that this is a review that tells us how the user feels about the Course Content.

Integrating MindsDB with Python

After following the installation guide in the backend repository README.md, we should have installed the required dependency for integrating MindsDB because it's included in the requirements.txt file in the repository.

This is the MindsDB Python SDK. To install it explicitly using pip, run the following command in the terminal.

pip install mindsdb-sdk

This is the program from the src/utils/base/mindsdb.py file in the repository.

import mindsdb_sdk
from django.conf import settings
from Curriculum.models import CurriculumReview

username = settings.MINDSDB_SERVER_USERNAME
password = settings.MINDSDB_SERVER_PASSWORD

server = None


def connect_server():
    global server
    if not server:
        server = mindsdb_sdk.connect(
            'https://cloud.mindsdb.com',
            login=username,
            password=password
        )
    return server


def make_query(text: str, model: str):
    return f"""
    SELECT *
    FROM mindsdb.{model}
    WHERE text = '{text}'
    """


def classify_text(text: str):
    server = connect_server()
    project = server.get_project('mindsdb')
    max_label = ""
    max_score = 0

    df = project.query(make_query(text, "hf_classify")).fetch()
    data = df['PRED_explain'][0]
    for k, v in CurriculumReview.LABEL:
        current = data.get(v)
        if current > max_score:
            max_score = current
            max_label = k

    return max_label


def sentiment_text(text: str):
    server = connect_server()
    project = server.get_project('mindsdb')
    max_label = ""
    max_score = 0

    df = project.query(make_query(text, "hf_sentiment")).fetch()
    data = df['PRED_explain'][0]
    for k, v in CurriculumReview.SENTIMENT:
        current = data.get(v)
        if current > max_score:
            max_score = current
            max_label = k
    return max_label

Let's go through this program 🤓

1. Importing the required modules

    import mindsdb_sdk
    from django.conf import settings
    from Curriculum.models import CurriculumReview
   

   We imported the CurriculumReview model so we can have access the labels defined in it.

   The Django setting is imported so we can have access to the MINDSDB_SERVER_USERNAME and MINDSDB_SERVER_PASSWORD defined when we set up the backend project.

2. Creating a connection to our MindsDB cloud server.

    server = None
   
    def connect_server():
        global server
        if not server:
            server = mindsdb_sdk.connect(
                'https://cloud.mindsdb.com',
                login=username,
                password=password
            )
        return server
   

   We did it this way so that if the same Django process already has a connection created for the mindsdb_sdk server, we will use it instead of creating it again.

3. Creating SQL model queries to our mindsdb project in the cloud

    def make_query(text: str, model: str):
        return f"""
        SELECT *
        FROM mindsdb.{model}
        WHERE text = '{text}'
        """
   

   This function takes the text to classify and the model name, hf_classify or hf_sentiment, and returns a query that we can execute.

4. Classifying text to label

    def classify_text(text: str):
        server = connect_server()
        project = server.get_project('mindsdb')
        max_label = ""
        max_score = 0
   
        df = project.query(make_query(text, "hf_classify")).fetch()
        data = df['PRED_explain'][0]
        for k, v in CurriculumReview.LABEL:
            current = data.get(v)
            if current > max_score:
                max_score = current
                max_label = k
   
        return max_label
   

   We first

   • get the connection to our MindsDB cloud

   • then get the project, mindsdb,

   • then use the project to make a query to the hf_classify model,

   • parse the response and get the label with the maximum result.

5. Classifying text to sentiment

   Works exactly like classify_text, we just used CurriculumReview.SENTIMENT when getting the label with the maximum result.

This is a program from /src/Curriculum/api/base/views.py for RateCurriculum view.

def perform_create(self, serializer):
    cur = self.get_object()
    enrollment = self.request.user\
        .get_curriculum_enrollment(cur)
    review = serializer.save(enrollment=enrollment)

    # Classify review here, in live app use cron job
    review.sentiment = sentiment_text(review.review)
    review.label = classify_text(review.review)
    review.save()

Ensuring the review label is updated when a user submits a new review, we classify it and update the database.

A performance problem here is that the user has to wait for our classification algorithm to work before it returns a response. In a production environment, it will be better to do the labeling later by using a Queue or Cron job that classifies reviews in the background.

Impact

By using MindsDB models, we are able to quickly and accurately classify user curriculum reviews, allowing us to identify and address areas for improvement. Using the MindsDb cloud made it easier to integrate AI into the application. Doing this the manual way will be hard and time-consuming.

This helps us to continually improve the platform and curate the best possible curriculums for users. Additionally, by using Django and Next.js, we are able to build a scalable and secure platform that can handle a large number of users and provide a seamless user experience.

Conclusion

In this tutorial, we showed how to use MindsDB models and Django for the backend to build a tech course curriculums platform. By implementing MindsDB models for sentiment and text classification, we can quickly and accurately classify user curriculum reviews and improve the user experience of your platform. With the power of MindsDB, we can build a scalable and secure platform that provides a seamless user experience.

Explore the features of MindsDB by checking out their official documentation and joining the Slack community.

Start building today by creating a free MindsDB Cloud account!

Want to Contribute?

This project is Open Source and can help a lot of new developers organize online free content into structured curriculums saving them from the Tutorial Hell.

Connect with me on Twitter | LinkedIn | GitHub, and let's work together, Thanks. 🤓💻

Prerequisites

You must install Python on your machine if you don't have it installed. It’s the only needed external requirement. You can install Python from their official downloads page. It’s recommended to install the latest version of Python for this tutorial.

To follow up with this tutorial, you are required to have some knowledge about writing simple Python programs. All the packages that will be discussed are from the Python standard library or Pypi package manager.

Secured Symmetrical Encryption Keys

When it comes to encryption and decryption in Python, the Pypi cryptography module is a popular choice. One of the easiest and most secure ways to implement symmetric cryptography using this module is through Fernet symmetrical encryption. However, to use Fernet keys for symmetric encryption, they must be generated and stored securely.

Let's dive in.

Start by creating a new Python project and set up your virtual environment to separate your project dependencies from your global dependencies.

python -m venv venv
# or
python3 -m venv venv

The Python command above is possible with Python3. If you have any errors setting this up, check out Python's official venv documentation to learn about creating a virtual environment.

• Activate the created environment

# Windows: cmd.exe
venv\Script\activate

# Linux: bash or Mac: zsh
source venv/bin/activate

Depending on your shell, the command for activating your virtual environment will be different. Check this reference for the command that will work for your shell.

• Now you need to install cryptography

pip install cryptography

If you encounter any installation issues, you can check out its installation guide.

Enter the Python shell and run the following commands to generate the key.

>>> from cryptography.fernet import Fernet
>>> key = Fernet.generate_key()
b'RAuFlYBGswBmBOccV13UNYXxJTi19LCGhUOLZOi6oFY='

The value RAuFlYBGswBmBOccV13UNYXxJTi19LCGhUOLZOi6oFY= in the byte string is the important key. It's stored like that because that's what the cryptography library likes to work with. So, if you generate a new key in the future, you need to convert it to bytes before passing it to any hash function.

Is Fernet.generate_key() cryptographically secure?

The answer is a resounding YES!

The implementation of Fernet.generate_key method uses the function os.urandom, which has more sufficient randomness than the random function from the built-in random module.

Why? It's because os.urandom cannot be seeded like random and draws its source of entropy from many unpredictable sources, making it more random. For more understanding, you can check this StackOverflow answer.

Now you can use your securely generated key to encrypt and decrypt data.

# Initialize a Fernet object
>>> f = Fernet(key)

# Generate a cryptographically secure and url-safe fernet token
>>> token = f.encrypt(b"Hashnode Neptune is the best")
>>> print(token)
b'gAAAAABkDv-R51WpztocZoMMat3UyKg8jz6KgCQgCq4g9SU36OF7kiPhqQwjLXPT-39lbb5cL-MlUWSmoDLKXlkOZo2Od_Icp_6jPFLDgF32f2r9agrRr50='

There you have it. You have generated a secure token that you can use to ensure your data security in your projects.

If you don't want to use the generate_key function to create your Fernet tokens. You can use this resource to generate a more secure and strong token using the PBKDF2HMAC algorithm.

Securely Managing Your Secret Keys

To ensure complete security, you must store your keys safely. Never share with anyone.

One easy way to do this is to use Pypi decouple.

• Let's start by installing it in our environment.

pip install python-decouple

• Create a file .env in your project directory. Add the key generated earlier to the file.

SECRET_KEY="RAuFlYBGswBmBOccV13UNYXxJTi19LCGhUOLZOi6oFY="

Save the file.

• Create a new Python program, main.py, in your project directory. Then paste the code added below into the file.

from decouple import config

SECRET_KEY = config('SECRET_KEY')

print(SECRET_KEY)

Run the main.py python program.

python main.py

The secret key stored in the environment file would be displayed. Decouple is a project used to store the environment and secret keys. You can read more at Pypi Decouple.

What makes this approach better than using an operating system environment? Because it makes it easier to share secrets when collaborating with others. They only need to clone the project and copy the .env file.

Also, it eliminates the collision of similar keys on multiple projects on the same machine. As a bonus, you can create a file env.example, this file will contain the keys from the actual .env file but with fake values.

Your collaborator can then;

1. Create their .env file,

2. Copy the contents of the env.example file into their .env file,

3. Generate a secret key,

4. Replace the sample (fake) key in the .env file with the newly generated key.

Ensure you don't forget to add .env file to .gitignore so it doesn't get pushed to your GitHub repository, thereby exposing your keys. It's better to have a sample file like env.example which can be pushed to keep track of what keys a collaborator needs to create when running the project on their machine.

Using Python Built-in Secrets Library

From Python Documentation: The secrets module is used for generating cryptographically strong random numbers suitable for managing data such as passwords, account authentication, security tokens, and related secrets.

Use the Secrets library for security purposes to generate something random, like random tokens, digits, or strings. Its randomness is cryptographically secure.

Creating a secure hex token

• Run your Python shell.

>>> import secrets
>>> secrets.token_hex()
'20a3f7333abd0668e474d393870a0b47463a6935e2eb730343820767eaf77226'

That's a simple way to create a very secure token. The longer the token the more sufficient the randomness.

The function token_hex accepts a parameter, n_bytes, for how long you want the token.

>>> import secrets
>>> secrets.token_hex(64
'5222cffc8c2881afcbf219c90c6e2f3a8b168c7547f790c2e852a047e9f4c8094577d8198a1d802ee053d13987e9111b317a7771c9ce87597a67203311afc69f')

Read more in the Python Documentation

Now you can use generate highly secure tokens for your projects. These tokens can be used as a secret key in any project, Django, NodeJS, Golang, etc.

Conclusion

Securing web applications and software requires effective management of secret keys. Python offers libraries like cryptography and secrets that simplify the process of generating cryptographically secure tokens. To ensure maximum protection of sensitive data, it is crucial to store these keys securely using tools like Pypi Decouple and AWS Secrets Manager. By adhering to best practices for managing secret keys, developers can guarantee the security of their projects and prevent unauthorized access to sensitive information.

Read more

1. Cryptography Algorithms - Primitives

2. Password Vaults

3. AWS Secrets Manager

Follow me on Twitter @netrobeweb, Hashnode, and Dev.to where I post amazing projects and articles.

Thanks for reading, 🤓😊

TL;DR

The system includes a registration and login page and once logged in, users will be greeted with a welcome voice. Follow the step-by-step guide to set up the backend API using Django and the frontend using Next.js and Chakra UI. The article also covers the prerequisites, project setup, installations, and building of the API. The API includes endpoints for Registration, Login, JWT Token Refresh, User Retrieve, and Speech to Text.

Prerequisites

Hey there! To make the most of this article, it would be helpful to have a solid understanding of Django, Django Rest Framework, and Next.js. Don't worry though, we'll do our best to explain things clearly and make it easy to follow along.

Getting Started

First of all, we would work on the backend. Django will be used for the Backend and Next.js + Charkra UI for the front end. Django is used because we would be using a Python package that helps us convert speech to text.

Application Design

The overall application design will be very simple. The backend will be an API which the frontend will consume.

1. A user's voice will be recorded on the frontend.

2. The frontend sends the recorded wav file to the backend in Base64 format.

3. The backend will process the data and convert it to text.

4. The text is returned to the frontend as a response,

5. then text is entered into the UI input box.

That's it, very simple.

Project Setup

This project is already available on GitHub. Clone it and follow the installation guide in the Readme to set up the project.

Installations

For building this project, we will need to have the following installed;

• Python 3.9 - This is used for the backend, and Django is used as the framework.

• NodeJs & NPM - This is used for the frontend.

Building the API

Let's clone the project.

git clone https://github.com/devvspaces/speech_to_text_auth
cd speech_to_text_auth

For now, we would be working on the API. Move into the api directory.

Set up your virtual environment

python -m venv venv
# Or python3 if you have that installed
python3 -m venv venv

Check out this python documentation to learn about virtual environments.

Activate the environment.

# windows: cmd.exe
venv\Scripts\activate

# bash or zsh
source venv/bin/activate

If the above commands don't work, check out this python documentation for help activating virtual environments. If you encounter an issue activating your virtual environment, here is a list of commands for common shells.

Installing requirements

# This will install the requirements in the requirements.txt file
pip install -r requirements.txt

Move into the src directory. There we have the account application, config, tests, and utils directories.

Now we need to create our .env file. This is where our secret configurations are kept. Decouple is used for accessing this file in our config files e.g manage.py, wsgi.py, etc.

Create a new file, .env and copy the contents of .env.example into it.

Now we need to create a Postgres DB for our project. By default, the project uses PostgreSQL but can be changed in the settings file. We can also create a DB for tests if we want to run tests. It's customary to have a separate database for development and testing because well testing uses autogenerated data which we don't want in our development database.

The config folder is where our settings are located. This directory contains a settings directory which contains two Python files, base.py and test.py. Having our settings file arranged like this helps us to write separate settings for different environments.

We could

• develop using the base.py,

• test using the test.py,

• maybe create a file staging.py for our staging environment in a live server

• and production.py for our production environment.

The tests folder contains our tests.

Accounts Application

This is the only application on the project and the most important because we will store users' data here. The API views, models, and serializers are designed here.

models.py

This is a basic Django custom user model implementation. By inheriting the Django AbstractBaseUser model, we can build our User model and make email required.

This is the DBML structure used for the diagram above.

// Use DBML to define your database structure
// Docs: https://www.dbml.org/docs
// DB Diagram tool: https://dbdiagram.io/home

Table users {
  pk integer [primary key]
  email varchar
  active boolean
  staff boolean
  admin boolean
  created timestamp
}

Table profile {
  pk integer [primary key]
  user_id integer
  fullname varchar
  sex varchar
  phone varchar
  country varchar
}

Ref: profile.user_id - users.pk // one-to-one

The User model has a one-to-one relationship with the Profile model. This is a very common way to design a User table. The reason for this is to make sure that when there is a change in business logic that will slightly affect how the User is created. We will update the Profile model instead so that we won't have migration issues with the User model.

When a user is created, we automatically create its profile by listening to the post_save signal. Learn more about Django Signals.

forms.py

We need a custom user registration form because we need a way to make the Django admin create a User correctly. This form allows the admin to enter the user's profile data when creating the user. The default Django admin form will have only fields for the User model.

By overriding the model form save method, we get the validated profile data and populate the already created user profile. As soon as the user.save() is invoked, the profile will be created by the post_save signal from earlier.

Learn more about creating Django Model Form.

admin.py

One important thing here is, we create a UserAdmin class for overriding the default that will be created by Django. Read more about Customizing Django Model Admin.

Django BaseUserAdmin is a special ModelAdmin that allows us to override the form used for creating a new User instance in the admin. So we pass the form we created earlier here.

config/settings/base.py

An important thing to note here is that after creating a Custom Django User Model we need to tell Django we want to use it instead of the default.

The line below tells Django to use our new Custom User Model instead of its default User Model. This is a very common implementation in Django, here is a Django Rest Framework GitHub template that we can use as a project boilerplate.

AUTH_USER_MODEL = 'account.User'

Decouple is used to access your environment keys, .env. This documentation shares how to effectively use decouple with Django.

Views and Serializers

Now we will cover the Django rest framework views and serializers. We need the following endpoints;

1. Registration [POST],

2. Login [POST],

3. JWT Token Refresh [POST],

4. User Retrieve [GET], and

5. Speech to Text [POST].

Registration [POST]: The endpoint will be used to create a new user. The data that will be received through all the endpoints will be in JSON format. The endpoint uses the imported RegisterSerializer to convert post data into native Python datatypes and validate them. An important thing to note here is that we are using JSON Web Token (JWT) for authenticating our users. As soon as a new account is created, new access and refresh tokens are included in the response. Read more about Simple JWT, A JSON Web Token authentication plugin for the Django REST Framework.

Login [POST]: Validates the user data and returns a response that contains the tokens and logged-in user data.

JWT Token Refresh [POST]: This is used to get a new access token using a valid refresh token. As JWT is a stateless authentication method, meaning there is no session stored for the user. Every 5 mins the access token previously generated is expired which requires the user to generate a new access token using the refresh token which is valid for 24 hours. The validation time for both the access and refresh tokens can be edited from the settings.py file. Learn more about this at Simple JWT.

User Retrieve [GET]: We need to be able to get the data of the currently authenticated user. This endpoint returns the data of the authenticated user.

Speech to Text [POST]: This endpoint used the SpeechRecognition package, to convert audio binary to text. This package supports Google, IBM, and Microsoft Speech-to-text APIs and many more. Here we are using Google Speech-to-text. Because of the abstraction, the package brings, we don't need to call any actual API to upload our audio files for processing.

class SpeechToTextView(APIView):

    @swagger_auto_schema(
        request_body=serializers.SpeechBody,
    )
    def post(self, request, *args, **kwargs):
        """
        Convert speech to text, using Google Speech Recognition API
        Accepts a base64 encoded string of audio data
        Returns:
            Response: Response object with text
        """
        try:
            record: str = self.request.data.get('record')
            decoded_b64 = base64.b64decode(record)

            # Convert decoded_b64 to in-memory bytes buffer
            r = sr.Recognizer()
            with sr.AudioFile(io.BytesIO(decoded_b64)) as source:
                # listen for the data (load audio to memory)
                audio_data = r.record(source)
                # recognize (convert from speech to text)
                text = r.recognize_google(audio_data)

            return Response(data={'text': text})
        except Exception:
            return Response(
                data={'message': "Error converting speech to text"},
                status=status.HTTP_400_BAD_REQUEST)

The data that will be passed to this endpoint will be in base64 format because it's faster and easier to transfer and process in this form.

decoded_b64 = base64.b64decode(record)

This takes the data received and decodes it using the built-in base64 library. The decoded data returned is in binary form. This is done so that we can convert it into a Buffered stream in memory using io.BytesIO(decoded_b64) , the built-in io library which is used for working with Input and Output (I/O) streams.

What are Buffered I/O streams in Python?

Buffered I/O streams in Python's io module are classes that help improve the performance of I/O operations by buffering data in memory. Think of buffering as a way of temporarily storing data in a queue so that it can be accessed more efficiently later on. Read more on Buffered streams to deeply understand how it works.

When working with objects like Buffers, Files, or Database connections in Python Context Managers are used to gain access to these objects because the connection is closed automatically even if there is an unhandled exception in the context manager.

with sr.AudioFile(io.BytesIO(decoded_b64)) as source:
    # listen for the data (load audio to memory)
    audio_data = r.record(source)
    # recognize (convert from speech to text)
    text = r.recognize_google(audio_data)

If an error occurs while trying to record the audio source, which is the Buffer we created earlier in the memory, the I/O connection will be closed safely. This article explains why it's important to use Context Managers when dealing with objects like this.

So the buffer is recorded and passed to google for speech-to-text processing. If all is well, we return the text in the response else we raise a BadRequest error.

Conclusion 🤓

In conclusion, this article provides a comprehensive tutorial on how to build a speech authentication system with Django. The system allows users to enter their details by speaking and includes a registration and login page. The article covers the step-by-step guide to setting up the backend API using Django. The article also covers the necessary installations, project setup, and the design of the application. Overall, this article is a great resource for developers looking to build a speech authentication system.

To make this article brief the next part which is Building the Frontend with Next.js will be covered in another article.

You can follow me on Twitter, Hashnode, Dev.to, and Github, where I post amazing projects and articles.

Thanks for reading, 😉.

If you don't know how to handle file upload in NestJs, this article explains how to efficiently incorporate file upload and validation into your NestJS project.

Creating a Cloudinary Module

One way to easily set it up in your NestJs projects is by creating a module for it. This allows you to use Dependency Injection design to use it in other providers or controllers.

Installation

Install Cloudinary NodeJs SDK

npm install cloudinary

Create a new module, cloudinary.

Create file constants.ts.

export const CLOUDINARY = 'Cloudinary';

Add file cloudinary.provider.ts, paste the code below into the file.

import { ConfigService } from '@nestjs/config';
import { v2 } from 'cloudinary';
import { CLOUDINARY } from './constants';

export const CloudinaryProvider = {
  provide: CLOUDINARY,
  useFactory: (config: ConfigService) => {
    return v2.config({
      cloud_name: config.get('CLOUDINARY_CLOUD_NAME'),
      api_key: config.get('CLOUDINARY_API_KEY'),
      api_secret: config.get('CLOUDINARY_API_SECRET'),
    });
  },
  inject: [ConfigService],
};

ConfigService is used to assess keys in your secret file. For example, a .env file.

Don't know how to set up env configurations in NestJs? Read more.

Don't know how to create factory providers in NestJs? Read more.

The CloudinaryProvider provider is a Factory Provider. The useFactory allows you to create providers dynamically. You can also inject other module-imported services into the useFactory function.

Create the Cloudinary Service Injectable

Add a new file, cloudinary.service.ts

import { Injectable } from '@nestjs/common';


@Injectable()
export class CloudinaryService {   
}

This service will contain methods for uploading files to Cloudinary.

Create the module file

Add a new file, cloudinary.module.ts

import { Module } from '@nestjs/common';
import { ConfigModule } from '@nestjs/config';
import { CloudinaryProvider } from './cloudinary.provider';
import { CloudinaryService } from './cloudinary.service';

@Module({
    imports: [ConfigModule],
    providers: [CloudinaryProvider, CloudinaryService],
    exports: [CloudinaryProvider, CloudinaryService],
})
export class CloudinaryModule {}

The ConfigModule is added to the module imports because ConfigService in the CloudinaryProvider.

Setup .env file

Create a .env in your root folder. Depending on how you structure your project, a general root folder could be the folder containing your package.json file or where you run the command to start your server.

CLOUDINARY_CLOUD_NAME=test
CLOUDINARY_API_KEY=test
CLOUDINARY_API_SECRET=test

Now you need to create a free Cloudinary account. After creating your account, you can follow this guide to get your API key, secret and cloud name.

Setup ConfigModule

In your app.module.ts you will need to configure the ConfigModule. This makes sure it finds the .env file and populate itself with the keys and values, thereby making you access them with ConfigService.

import {
  Module,
  NestModule,
} from '@nestjs/common';
import { AppController } from './app.controller';
import { AppService } from './app.service';
import { ConfigModule } from '@nestjs/config';

@Module({
  imports: [
    ConfigModule.forRoot(),
  ],
  controllers: [AppController],
  providers: [
    AppService,
  ],
})
export class AppModule implements NestModule {
}

Using Form data - File Buffers

Single File Upload

Now that you have your CloudinaryService. When you create a route that accepts file buffers using Multer. You can upload this file using the service. You just have to integrate the upload method from the Cloudinary package.

Add the method below to your service.

async uploadFile(
  file: Express.Multer.File,
): Promise<UploadApiResponse | UploadApiErrorResponse> {
  return new Promise((resolve, reject) => {
    v2.uploader.upload_stream(
      {
        resource_type: 'auto',
      },
      (error, result) => {
        if (error) return reject(error);
        resolve(result);
      }
    ).end(file.buffer)
  })
}

This simple method uses v2.uploader.upload_stream to upload your file to Cloudinary in chunks, which is good for uploading large files in chunks instead of one long request.

An UploadApiResponse or UploadApiErrorResponse is returned by the function. You can use this to know if the file was successfully uploaded or not.

To access your uploaded file URL, use the example of a route for single file upload below to understand the process.

import { UploadedFile, UseInterceptors } from '@nestjs/common';
import { FileInterceptor } from '@nestjs/platform-express';

@Post('upload/single')
@UseInterceptors(FileInterceptor('file'))
@ApiOkResponse({ description: 'Upload image', type: String })
public async uploadSingleImage(@UploadedFile() file: Express.Multer.File) {
  const result = await this.cloudinaryService.uploadFile(file);
  return result.secure_url;
}

The secure_url property is the URL for the file uploaded.

Uploading multiple file buffers

Loop through all the files and upload each one after the other. The response is the list of secure URLs.

async uploadFiles(
  files: Express.Multer.File[],
) {
  const urls = await Promise.all(files.map(async (file): Promise<string> => {
    const { secure_url } = await this.uploadFile(file);
    return secure_url;
  }));
  return urls
}

The files were mapped over and uploaded, and their URLs were returned by the map callback function.

Uploading File with URLs

What if you have a file URL and not a file buffer? It's simpler to upload that using Cloudinary.

async uploadFileFromUrl(
  url: string,
): Promise<UploadApiResponse | UploadApiErrorResponse> {
  return v2.uploader.upload(url)
}

The URL passed to this would be uploaded to Cloudinary, which will then return a response containing the secure_url if successful.

Uploading multiple file urls

By following the same idea from uploading multiple file buffers. You can map through the urls, upload each and return the secure_url for each. Then the function will return a list of URLs.

async uploadFilesFromUrl(
  urls: string[],
) {
  return Promise.all(urls.map(async (url: string): Promise<string> => {
    const { secure_url } = await this.uploadFileFromUrl(url);
    return secure_url;
  }));
}

Uploading Base64 encoded strings

Base64 is a binary-to-text encoding scheme that represents binary data in an American Standard Code for Information Interchange (ASCII) string format. An example of binary data is an image. Base64 can be uploaded too as a file, it's a string and not a buffer, which makes it easier to parse and upload to Cloudinary than file buffers.

By utilizing the upload function from the Cloudinary package, a base64 string can be uploaded to Cloudinary and get a URL for the uploaded file.

async uploadFileFromBase64(
  data: string,
): Promise<UploadApiResponse | UploadApiErrorResponse> {
  return v2.uploader.upload(data)
}

async uploadManyBase64(
  files: string[],
) {
  const urls = await Promise.all(files.map(async (file): Promise<string> => {
    const { secure_url } = await this.uploadFileFromBase64(file);
    return secure_url;
  }));
  return urls
}

Conclusion

In conclusion, Cloudinary is a powerful tool for file uploading and storage in NestJS projects. With the help of the Cloudinary NodeJS SDK and the steps outlined in this article, developers can easily set up a module for Cloudinary and upload files using various available methods including file buffers, file URLs, and Base64 encoded strings. While Cloudinary is free to set up initially, developers may need to pay for used services once they start using it past some limit.

I'd love to connect ❤️ with you on Twitter | LinkedIn | GitHub | Portfolio

See you in my next blog article. Take care!!!

Moreover, AdminJS offers customization options that enable you to personalize the dashboard according to your project requirements. Kindly read further to acquire guidance on initiating AdminJS and attaining mastery over your application's data management.

What is ORM

Object-relational mapping (ORM) in computer science is a programming technique for converting data between incompatible type systems using object-oriented programming languages.

What is ODM?

Object Document Mapper (ODM) is a library that helps you to work with MongoDB databases. It provides an abstraction layer between the database and the application code. It allows you to use the object-oriented paradigm to work with the database.

Getting Started

AdminJs + NestJs => AdminJs + Prisma => Customizations

The aforementioned flow provides a concise overview of the process involved in configuring AdminJs for your NestJs project.

The recommended approach entails commencing the process by installing the requisite AdminJs packages. Subsequently, configure AdminJs with NestJs followed by Prisma, and finally, incorporate customizations as necessary which is optional.

Install Requirements

It's required that you have your nest project already set up, if you don't know how to create one you can use the official NestJs documentation.

yarn add adminjs @adminjs/nestjs @adminjs/express express-session express-formidable

All the requirements above are required by AdminJs and more information on them can be provided by AdminJs Docs.

app.module.ts

import { Module } from '@nestjs/common'
import { AdminModule } from '@adminjs/nestjs'

import { AppController } from './app.controller'
import { AppService } from './app.service'

const DEFAULT_ADMIN = {
  email: 'admin@example.com',
  password: 'password',
}

const authenticate = async (email: string, password: string) => {
  if (email === DEFAULT_ADMIN.email && password === DEFAULT_ADMIN.password) {
    return Promise.resolve(DEFAULT_ADMIN)
  }
  return null
}

@Module({
  imports: [
    AdminModule.createAdminAsync({
      useFactory: () => ({
        adminJsOptions: {
          rootPath: '/admin',
          resources: [],
        },
        auth: {
          authenticate,
          cookieName: 'adminjs',
          cookiePassword: 'secret'
        },
        sessionOptions: {
          resave: true,
          saveUninitialized: true,
          secret: 'secret'
        },
      }),
    }),
  ],
  controllers: [AppController],
  providers: [AppService],
})
export class AppModule {}

You might want your admin panel to not require authentication depending on the project you are working on. By removing the auth and sessionOptions from the object returned by the factory provider.

app.module.ts

...
AdminModule.createAdminAsync({
  useFactory: () => ({
    adminJsOptions: {
      rootPath: '/admin',
      resources: [],
    },
  }),
}),
...

The upper part and lower part of the app.module.ts is removed to conserve space.

Start your server

nest start --watch
# OR
nest start

Visit 127.0.0.1:<your port>/admin . If you have authentication enabled it will automatically redirect you to the login page.

Enter the details from the DEFAULT_ADMIN object in the app.module.ts program.

Using Adapters to Connect to the Database in AdminJS

This tutorial focuses on using Prisma as the ORM. Prisma is open-source, has an auto-generated type-safe client for Node.js & Typescript, and manages migrations. AdminJs also work with other ORMs e.g TypeORM etc. They are called adapters and you can choose what you want to use. If you want to learn Prisma, follow this guide to getting started with Prisma ORM.

If you can't find your models on the dashboard after logging in, this is because AdminJs uses adapters you defined to connect to the database. It uses it as an intermediary to perform database actions like querying, creating, updating, etc.

schema.prisma

generator client {
  provider = "prisma-client-js"
}

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

model User {
  pk            Int      @id @default(autoincrement())
  id            String   @unique @default(uuid()) @db.Uuid
  email         String   @unique
  password      String
  active        Boolean  @default(true)
  staff         Boolean  @default(false)
  admin         Boolean  @default(false)
  profile       Profile?

  @@unique([pk, id])
  @@map("users")
}

model Profile {
  pk              Int                      @id @default(autoincrement())
  id              String                   @unique @default(uuid()) @db.Uuid
  firstName       String?                  @map("first_name")
  lastName        String?                  @map("last_name")
  userPk          Int                      @unique @map("user_pk")
  userId          String                   @unique @map("user_id") @db.Uuid
  user            User                     @relation(fields: [userPk, userId], references: [pk, id], onDelete: Cascade, onUpdate: Cascade)

  @@unique([pk, id])
  @@unique([userPk, userId])
  @@map("profiles")
}

app.module.ts

import { Module } from '@nestjs/common'
import { AdminModule } from '@adminjs/nestjs'
import { Database, Resource } from '@adminjs/prisma';
import AdminJS from 'adminjs'
import { DMMFClass } from '@prisma/client/runtime';
import { PrismaService } from '@database/prisma.service';
import { DatabaseModule } from '@database/database.module';

import { AppController } from './app.controller'
import { AppService } from './app.service'

const DEFAULT_ADMIN = {
  email: 'admin@example.com',
  password: 'password',
}

const authenticate = async (email: string, password: string) => {
  if (email === DEFAULT_ADMIN.email && password === DEFAULT_ADMIN.password) {
    return Promise.resolve(DEFAULT_ADMIN)
  }
  return null
}

AdminJS.registerAdapter({ Database, Resource });

@Module({
  imports: [
    AdminModule.createAdminAsync({
      imports: [DatabaseModule],
      useFactory: (prisma: PrismaService) => {
        const dmmf = (prisma as any)._baseDmmf as DMMFClass;

        return {
          adminJsOptions: {
            rootPath: '/admin',
            resources: [
              {
                resource: { model: dmmf.modelMap['User'], client: prisma },
                options: {},
              },
              {
                resource: { model: dmmf.modelMap['Profile'], client: prisma },
                options: {},
              },
            ],
          },
          auth: {
            authenticate,
            cookieName: 'adminjs',
            cookiePassword: 'secret'
          },
          sessionOptions: {
            resave: true,
            saveUninitialized: true,
            secret: 'secret'
          },
        }
      },
      inject: [PrismaService],
    }),
  ],
  controllers: [AppController],
  providers: [AppService],
})
export class AppModule {}

Explore the basic Resource object format and effortlessly add multiple resources to your project.

{
  resource: { model: dmmf.modelMap[{Model name here}], client: prisma },
  options: {},
}

Make sure you replace {Model name here} with the name of the model you want to use. For example, if you have a model Post, you would replace {Model name here} with Post.

How does this work?

The Database Model Mapper (DMMF) object comprises a comprehensive depiction of your database schema. Its functionality extends to facilitating the extraction of your model from Prisma. The DMMF achieves this by mapping model names to their respective model objects. For example, to retrieve the User model from the map, one can utilize the syntax dmmf.modelMap['User']. Once the model object is obtained, it can be assigned to the resource's model property.

The Prisma client serves as the client property, facilitating connectivity to the database for conducting CRUD operations.

You can add as many resources as you need. For example, if you have a Post model in your Prisma schema that you want to be displayed in the Admin Dashboard. You just need to add it to the resources array;

app.module.ts

...
{
  resource: { model: dmmf.modelMap['Post'], client: prisma },
  options: {},
},
...

Now restart your server if it's not on fast reload.

Simplify adding resources

When managing a substantial number of database models, it is common to encounter the need for duplicating codes while incorporating new resources for each model.

The implementation of the Builder pattern in coding allows for the addition of multiple resources without duplicating code, thereby enhancing code efficiency and ease of maintenance. Consider incorporating this pattern into your coding practices to reap the benefits.

import { Module } from '@nestjs/common'
import { AdminModule } from '@adminjs/nestjs'
import { Database, Resource } from '@adminjs/prisma';
import AdminJS from 'adminjs'
import { DMMFClass } from '@prisma/client/runtime';
import { PrismaService } from '@database/prisma.service';
import { DatabaseModule } from '@database/database.module';
import { ResourceWithOptions, ResourceOptions } from 'adminjs/types/src';

import { AppController } from './app.controller'
import { AppService } from './app.service'

const DEFAULT_ADMIN = {
  email: 'admin@example.com',
  password: 'password',
}

const authenticate = async (email: string, password: string) => {
  if (email === DEFAULT_ADMIN.email && password === DEFAULT_ADMIN.password) {
    return Promise.resolve(DEFAULT_ADMIN)
  }
  return null
}

AdminJS.registerAdapter({ Database, Resource });

class CResource {
  model: any;
  options: ResourceOptions;

  constructor(model: any, options?: ResourceOptions) {
    this.model = model;
    this.options = options || {};
  }
}

class CResourceBuilder {
  private readonly resources: Array<CResource> = [];
  dmmf: DMMFClass;

  constructor(private readonly prisma: PrismaService) {
    this.dmmf = ((prisma as any)._baseDmmf as DMMFClass)
  }

  /**
   * Adds a resource to the builder
   * 
   * @param resource string
   * @param options ResourceOptions
   * @returns this
   */
  public addResource(resource: string, options?: ResourceOptions): this {
    const obj = new CResource(this.dmmf.modelMap[resource], options);
    this.resources.push(obj);
    return this;
  }

  /**
   * Compiles the resources into an array of objects
   * that can be passed to the AdminJS module
   * 
   * @returns Array<ResourceWithOptions | any>
   */
  public build(): Array<ResourceWithOptions | any> {
    return this.resources.map((resource) => {
      return {
        resource: {
          model: resource.model,
          client: this.prisma,
        },
        options: resource.options,
      }
    })
  }
}

@Module({
  imports: [
    AdminModule.createAdminAsync({
      imports: [DatabaseModule],
      useFactory: (prisma: PrismaService) => {
        const dmmf = (prisma as any)._baseDmmf as DMMFClass;

        return {
          adminJsOptions: {
            rootPath: '/admin',

            // updated here
            resources: new CResourceBuilder(prisma)
              .addResource('User')
              .addResource('Profile')
              .addResource('Post')
              .build(),
          },
          auth: {
            authenticate,
            cookieName: 'adminjs',
            cookiePassword: 'secret'
          },
          sessionOptions: {
            resave: true,
            saveUninitialized: true,
            secret: 'secret'
          },
        }
      },
      inject: [PrismaService],
    }),
  ],
  controllers: [AppController],
  providers: [AppService],
})
export class AppModule { }

You can always export the classes, functions and objects from an external file.

Conclusion

In conclusion, AdminJS is a powerful tool for creating admin dashboards for NestJS and Prisma applications. With its adapters for various ORM and ODM models, it simplifies the process of performing CRUD operations on models. It also provides customization options for the dashboard, making it easy to tailor it to specific project needs. By following the steps outlined in the article, developers can quickly set up AdminJS and start using it to manage their application's data.

Resources

• More options for resources

• Customizing your Admin Dashboard

I'd love to connect with you on Twitter | LinkedIn | GitHub | Portfolio

See you in my next blog article. Take care!!!

Accepting files

File processing in NestJs is handled with Multer. It has a built-in Multer middleware package for Express Adapter. The only required package to install is @types/multer, which is for better type safety.

npm i -D @types/multer @nestjs/platform-express

Start by creating an endpoint for single file uploads.

import { Post, UploadedFile, UseInterceptors, Controller } from '@nestjs/common';
import { FileInterceptor } from '@nestjs/platform-express';

@Controller('files')
export class FileController {
    @Post('upload')
    @UseInterceptors(FileInterceptor('file'))
    uploadFile(@UploadedFile() file: Express.Multer.File) {
        console.log(file);
    }
}

Make sure to use @nestjs/common: "^9.0.0", it's required to use UploadedFile.

If you use swagger, you can specify ApiConsumes decorator to multipart/form-data. This tells swagger that this route accepts form data.

import { Post, UploadedFile, UseInterceptors, Controller } from '@nestjs/common';
import { FileInterceptor } from '@nestjs/platform-express';
import { ApiConsumes, ApiTags } from '@nestjs/swagger';

@ApiTags('Uploding Files')
@Controller('files')
@ApiConsumes('multipart/form-data')
export class FileController {
    @Post('upload')
    @UseInterceptors(FileInterceptor('file'))
    uploadFile(@UploadedFile() file: Express.Multer.File) {
        console.log(file);
    }
}

When it comes to filing storage, Multer has a default memory setting. While this is a viable choice, it may not suit your needs if you plan to bypass disk storage and instead upload files directly to external storage solutions like Cloudinary or AWS S3.

How to Save Data to Disk Storage

To upload your files to disk storage, you can provide the file destination in the FileInterceptor. The interceptor accepts Multer Options.

const UPLOAD_DIR = './upload/files/';

@ApiTags('Uploding Files')
@Controller('files')
@ApiConsumes('multipart/form-data')
export class FileController {
    @Post('upload')
    @UseInterceptors(FileInterceptor('file', { dest: UPLOAD_DIR }))
    uploadFile(@UploadedFile() file: Express.Multer.File) {
        console.log(file);
    }
}

The upload directory path is relative to your root directory.

Hey there! When dealing with a lot of users, we want to make sure we avoid any issues that might come up with file names. To prevent this from happening, we recommend assigning a separate folder for each upload. That way, everything stays organized and everyone can easily find what they need.

There is a helpful tip to share with you. Are you aware that every user can have their unique folder in the upload directory? What's more, you can enhance this feature by configuring a storage engine for Multer, which allows you to create a dynamic upload path. Why not give it a try?

import { Post, UploadedFile, UseInterceptors, Controller } from '@nestjs/common';
import { FileInterceptor } from '@nestjs/platform-express';
import { ApiConsumes, ApiTags } from '@nestjs/swagger';
import { Request } from 'express';
import { diskStorage } from 'multer';
import * as path from 'path';

const UPLOAD_DIR = './upload/files/';

// User interface of the authenticated user
interface User {
  id: string;
}

/**
 * You can use this function to generate a unique filename for each file
 * User id is used to generate a unique filename
 * The User object can be attached to the request object in the auth middleware
 */
const defaultConfig = diskStorage({
  destination: UPLOAD_DIR,
  filename: (req: Request & { user: User }, file, cb) => {
    const uid = req.user.id;
    cb(null, `${uid}${path.extname(file.originalname)}`)
  }
})

@ApiTags('Uploding Files')
@Controller('files')
@ApiConsumes('multipart/form-data')
export class FileController {

  @Post('upload')
  @UseInterceptors(FileInterceptor('file', { storage: defaultConfig }))
  uploadFile(@UploadedFile() file: Express.Multer.File) {
    console.log(file);
  }

}

In this example, the user object could be attached to the request manually in the auth middleware.

This is a perfect example of how to accept files and save them in a super cool and unique path!

The file would be saved to disk before console.log(file) runs.

The next section provides a way of adding file processing like validation of files uploaded.

Validate Your Files with ParseFilePipe and ParseFilePipeBuilder

Using ParseFilePipe with Validator classes

NestJs Pipes offer an efficient way to validate files. To perform file validation, simply create a validator class such as MaxFileSizeValidator or FileNameValidator, and pass it to the ParseFilePipe.

import { ..., ParseFilePipe } from '@nestjs/common';

...

@Post('upload')
@UseInterceptors(FileInterceptor('file', { storage: defaultConfig }))
uploadFile(
  @UploadedFile(
    new ParseFilePipe({
      validators: [
        // ... Set of file validator instances here
      ]
    })
  )
  file: Express.Multer.File
) {
  console.log(file);
}

...

Create a FileValidator class to use with ParseFilePipe.

import { FileValidator } from '@nestjs/common';

class MaxFileSize extends FileValidator<{ maxSize: number }>{
  constructor(options: { maxSize: number }) {
    super(options)
  }

  isValid(file: Express.Multer.File): boolean | Promise<boolean> {
    const in_mb = file.size / 1000000
    return in_mb <= this.validationOptions.maxSize
  }
  buildErrorMessage(): string {
    return `File uploaded is too big. Max size is (${this.validationOptions.maxSize} MB)`
  }
}

The interfaceFileValidator has two methods needed to create its class. The method isValid and buildErrorMessage.

The method, isValid will contain your logic that defines if the file is valid or not depending on the options passed in the constructor.

From the constructor definition, maxSize can be passed as an option and is used in the isValid method to know what is considered the maximum file size.

/**
 * Builds an error message in case the validation fails.
 * @param file the file from the request object
 */
abstract buildErrorMessage(file: any): string;

Above is the interface of the buildErrorMessage function. It's used to build unique or general error messages for uploaded files.

Pass an instance of the MaxFileSize class in the ParseFilePipe. The maxSize can be passed to the constructor during instantiation.

...

const MAX_UPLOAD_SIZE = 10; // in MB
...

@Post('upload')
@UseInterceptors(FileInterceptor('file', { storage: defaultConfig }))
uploadFile(
  @UploadedFile(
    new ParseFilePipe({
      validators: [
        // ... Set of file validator instances here
        new MaxFileSize({
          maxSize: MAX_UPLOAD_SIZE
        }),
      ]
    })
  )
  file: Express.Multer.File
) {
  console.log(file);
}

...

The code should look like this now

import { Post, UploadedFile, UseInterceptors, Controller, ParseFilePipe } from '@nestjs/common';
import { FileInterceptor } from '@nestjs/platform-express';
import { ApiConsumes, ApiTags } from '@nestjs/swagger';
import { Request } from 'express';
import { diskStorage } from 'multer';
import * as path from 'path';
import { FileValidator } from '@nestjs/common';

const UPLOAD_DIR = './upload/files/';
const MAX_UPLOAD_SIZE = 10; // in MB


// User interface of the authenticated user
interface User {
  id: string;
}


class MaxFileSize extends FileValidator<{ maxSize: number }>{
  constructor(options: { maxSize: number }) {
    super(options)
  }

  isValid(file: Express.Multer.File): boolean | Promise<boolean> {
    const in_mb = file.size / 1000000
    return in_mb <= this.validationOptions.maxSize
  }
  buildErrorMessage(): string {
    return `File uploaded is too big. Max size is (${this.validationOptions.maxSize} MB)`
  }
}

/**
 * You can use this function to generate a unique filename for each file
 * User id is used to generate a unique filename
 * The User object can be attached to the request object in the auth middleware
 */
const defaultConfig = diskStorage({
  destination: UPLOAD_DIR,
  filename: (req: Request & { user: User }, file, cb) => {
    const uid = req.user.id;
    cb(null, `${uid}${path.extname(file.originalname)}`)
  }
})

@ApiTags('Uploding Files')
@Controller('files')
@ApiConsumes('multipart/form-data')
export class FileController {

  @Post('upload')
  @UseInterceptors(FileInterceptor('file', { storage: defaultConfig }))
  uploadFile(
    @UploadedFile(
      new ParseFilePipe({
        validators: [
          // ... Set of file validator instances here
          new MaxFileSize({
            maxSize: MAX_UPLOAD_SIZE
          }),
        ]
      })
    )
    file: Express.Multer.File
  ) {
    console.log(file);
  }

}

If your validators are getting much, you can create them in a separate file and import them here as a named constant like myValidators. - NestJs Docs

Using ParseFilePipeBuilder

Wow, this is amazing! It's so much better because it comes with built-in validators for file size and type! Plus, there's a way to add even more validators using the addValidator method for complex file validation! How cool is that?!

import { ..., ParseFilePipeBuilder, HttpStatus } from '@nestjs/common';
...

@Post('upload')
@UseInterceptors(FileInterceptor('file', { storage: defaultConfig }))
uploadFile(
  @UploadedFile(
    new ParseFilePipeBuilder()
      .addFileTypeValidator({
        fileType: /(jpg|jpeg|png|gif)$/,
      })
      .addMaxSizeValidator({
        maxSize: 1000
      })
      .addValidator(
        new MaxFileSize({
          maxSize: MAX_UPLOAD_SIZE
        }),
      ) 
      .build({
        errorHttpStatusCode: HttpStatus.UNPROCESSABLE_ENTITY
      })
  )
  file: Express.Multer.File
) {
  console.log(file);
}

...

errorHttpStatusCode is the HTTP status code you want to be returned when validation fails.

Simplify the process of file validation with the help of ParseFilePipeBuilder. For straightforward validations like file types and sizes, this tool is highly recommended. However, for more intricate validations, it's best to create file validator classes.

Optional File Uploads

File upload parameters can be made optional or not required by passing an extra option in the ParseFilePipeBuilder build method.

new ParseFilePipeBuilder()
.addFileTypeValidator({
  fileType: "png",
})
.build({
  fileIsRequired: false  // It's required by default
})

Accepting multiple files

Simply change the file interceptor and upload the file decorator.

import { UploadedFiles } from '@nestjs/common';
import { FilesInterceptor } from '@nestjs/platform-express';

const MAX_FILES_COUNT = 10;

...

@Post('upload/multiple')
@UseInterceptors(
  FilesInterceptor('files', MAX_FILES_COUNT, { storage: defaultConfig })
)
uploadFiles(
  @UploadedFiles(
    new ParseFilePipeBuilder()
      .addFileTypeValidator({
        fileType: /(jpg|jpeg|png|gif)$/,
      })
      .addMaxSizeValidator({
        maxSize: 1000
      })
      .addValidator(
        new MaxFileSize({
          maxSize: MAX_UPLOAD_SIZE
        }),
      ) 
      .build({
        errorHttpStatusCode: HttpStatus.UNPROCESSABLE_ENTITY
      })
  )
  files: Express.Multer.File[]
) {
  console.log(files);
}

...

It's now UploadedFiles and FilesInterceptor. FilesInterceptor accepts max files number that can be uploaded through the endpoint.

Make sure to accept files as an argument, not just file.

Below, you can find the complete code.

import { Post, UploadedFile, UseInterceptors, Controller, ParseFilePipe, ParseFilePipeBuilder, HttpStatus, UploadedFiles } from '@nestjs/common';
import { FileInterceptor, FilesInterceptor } from '@nestjs/platform-express';
import { ApiConsumes, ApiTags } from '@nestjs/swagger';
import { Request } from 'express';
import { diskStorage } from 'multer';
import * as path from 'path';
import { FileValidator } from '@nestjs/common';

const UPLOAD_DIR = './upload/files/';
const MAX_UPLOAD_SIZE = 10; // in MB
const MAX_FILES_COUNT = 10; // Maximum number of files that can be uploaded at once


// User interface of the authenticated user
interface User {
  id: string;
}


class MaxFileSize extends FileValidator<{ maxSize: number }>{
  constructor(options: { maxSize: number }) {
    super(options)
  }

  isValid(file: Express.Multer.File): boolean | Promise<boolean> {
    const in_mb = file.size / 1000000
    return in_mb <= this.validationOptions.maxSize
  }
  buildErrorMessage(): string {
    return `File uploaded is too big. Max size is (${this.validationOptions.maxSize} MB)`
  }
}

/**
 * You can use this function to generate a unique filename for each file
 * User id is used to generate a unique filename
 * The User object can be attached to the request object in the auth middleware
 */
const defaultConfig = diskStorage({
  destination: UPLOAD_DIR,
  filename: (req: Request & { user: User }, file, cb) => {
    const uid = req.user.id;
    cb(null, `${uid}${path.extname(file.originalname)}`)
  }
})

@ApiTags('Uploding Files')
@Controller('files')
@ApiConsumes('multipart/form-data')
export class FileController {

  @Post('upload')
  @UseInterceptors(FileInterceptor('file', { storage: defaultConfig }))
  uploadFile(
    @UploadedFile(
      new ParseFilePipeBuilder()
        .addFileTypeValidator({
          fileType: /(jpg|jpeg|png|gif)$/,
        })
        .addMaxSizeValidator({
          maxSize: 1000
        })
        .build({
          errorHttpStatusCode: HttpStatus.UNPROCESSABLE_ENTITY
        })
    )
    file: Express.Multer.File
  ) {
    console.log(file);
  }

  @Post('upload/multiple')
  @UseInterceptors(
    FilesInterceptor('files', MAX_FILES_COUNT, { storage: defaultConfig })
  )
  uploadFiles(
    @UploadedFiles(
      new ParseFilePipeBuilder()
        .addFileTypeValidator({
          fileType: /(jpg|jpeg|png|gif)$/,
        })
        .addValidator(
          new MaxFileSize({
            maxSize: MAX_UPLOAD_SIZE
          }),
        )
        .build({
          errorHttpStatusCode: HttpStatus.UNPROCESSABLE_ENTITY
        })
    )
    files: Express.Multer.File[]
  ) {
    console.log(files);
  }
}

For a comprehensive configuration of Multer file upload, refer to the NestJs File Upload Documentation. Read on to learn more.

Conclusion

In conclusion, this article provides a comprehensive guide on how to implement file upload and validation in NestJS using the Multer middleware package for the Express Adapter. It covers accepting and validating files, saving files to disk storage, file validation with ParseFilePipe and ParseFilePipeBuilder, and accepting multiple files. By following the steps outlined in this article, developers can easily design their custom file upload and validation flow in NestJS.

To make this article as simple as possible, the definitions and terminologies would be summarized to just key points needed for the project you will build in this article. But links to amazing educational articles will be provided for each.

By the end of this article, you would understand Public Key Cryptography, Elliptic curve digital signatures and how they can be used to make the world a better place. I will take you one step at a time on how to build a blockchain node.

What is Cryptography

Cryptography is a way of securing a piece of information. It's the study of cryptographic hash functions.

Cryptographic hash functions are mathematical functions that take an input of any size and convert it to a fixed-sized output. It's a one-way ticket, you can't get the input from the output.

Cryptography is at the heart of blockchain and Web3. Here is an online tool containing lots of hash functions you can play with.

Public Key Cryptography

It's the use of two keys, a public key (a key that is known to everyone), and a private key (a key that is kept secured). The private key can encrypt a message that can only be decrypted by the public key, verifying that the message was indeed sent by someone with the right private key and vice versa.

Public key cryptography is also called asymmetric cryptography because it uses a pair of related keys.

RSA and ECDSA are two popular algorithms used for public cryptography. RSA is based on the idea that you can quickly find the product of two prime numbers, but extremely hard to factor out the prime numbers once you have the products. Read more about RSA.

ECDSA (Elliptic curve digital signature algorithm) uses elliptic curves. Read more about it on Wikipedia.

Why use ECDSA

ECDSA provides the same level of security as RSA but with small key sizes. RSA keys can be very large making it take a long to transfer over a network.

ECDSA is the algorithm used by Bitcoin, which is the secp256k1 curve.

Building an ECDSA node

This is an example of a blockchain node that uses the ECDSA signing algorithm. The project is a simple react frontend application that allows users to send money to each other. There will be a single server node, which makes this centralized but there will be an article later on how to deploy a program on Ethereum.

Goal

The server would manage the balances of the users. The goal is to enable the server to verify who is sending the money to make sure that a person can only send money from their wallet.

Project prerequisites

You will need to have a basic knowledge of React as it is used for the front end. With a basic understanding of Express.

Application Architecture

There is the front end, which has the UI part of the project that allows users to enter their wallet addresses and attempt to send money to others. The Idea here is that, when a user tries to send money, we would generate a transaction for them to sign with their private key, then input the signed transaction which would be sent to the server with their public key (wallet address), address where money should be sent, and the amount of money to be sent.

The server would take this signed transaction and the public key, then verify if the transaction was indeed signed using the right private key of the public key. If it's correct, it updates the users (sender and recipient) balances, else it returns an error message.

Client-Side React App

We would use Vite for our front-end tooling.

Create a new directory called wallet-node and move into the directory.

Run npm create vite@latest, enter client as your project name, react as the framework and JavaScript as the variant.

Move into the client directory just created by Vite. Run npm i to install the packages from the template.

Install Sass as a dev dependency using this command npm i -D sass. This will install the sass package as we will be using sass for our CSS styling.

The last installments are:

1. Axios - npm i axios

2. Ethereum Cryptography - npm i ethereum-cryptography

Now, edit the value of the title tag in the index.html file to ECDSA Wallet or anything you want.

Move into the src directory, delete the assets directory, index.css and App.css files.

Edit the main.jsx file, remove the line for import './index.css'.

Edit App.jsx by removing everything in it and pasting in the following code;

import Wallet from "./Wallet";
import Transfer from "./Transfer";
import "./App.scss";
import { useState } from "react";

function App() {
  const [balance, setBalance] = useState(0);
  const [address, setAddress] = useState("");

  return (
    <div className="app">
      <Wallet
        balance={balance}
        setBalance={setBalance}
        address={address}
        setAddress={setAddress}
      />
      <Transfer setBalance={setBalance} address={address} />
    </div>
  );
}

export default App;

We will create a Wallet and Transfer component, and our App.scss file for the sass styles.

From the above code, two stateful values were created, balance and address. The balance will be used to keep track of and manage an address balance, while the address will be used to share a user address among the Wallet and Transfer components.

The transfer component accepts a prop setBalance which would be used to set the state of the user's balance after the money has been transferred successfully.

Create an App.scss file in the src directory, then paste the following code;

body {
  font-family: "Muli", sans-serif;
  font-weight: 300;
  background-color: #e2e8f0;
  padding: 40px;
}

label {
  display: flex;
  flex-direction: column;
  letter-spacing: 0.05rem;
  font-size: .8em;
  font-weight: 400;
  color: #222;
}

.app {
  display: flex;
  max-width: 1400px;
  flex-wrap: wrap;
  gap: 12px;
  margin: 0 auto;
}

.container {
  flex-grow: 1;
  margin: 0 20px;
  background-color: #fff;
  border: 1px solid #cbd5e0;
  box-shadow: 0 1px 2px 0 rgba(0, 0, 0, 0.05);
  border-radius: 0.375rem;
  padding: 40px;

  label {
    margin: 10px 0;
  }

  .button {
    margin-top: 10px;
  }
}

input {
  padding: 10px 0;
  border-radius: 0.125rem;
  border: 1px solid rgb(226, 232, 240);
  background-color: #fdfdfe;
  padding-inline-start: 0.75rem;
  font-size: 0.875rem;
}

.button {
  background-color: #319795;
  border-radius: 0.125rem;
  padding: 10px 20px;
  color: white;
  display: inline-flex;
  text-transform: uppercase;
  letter-spacing: 1px;
  font-weight: 400;
  font-size: .9em;
  &:hover {
    cursor: pointer;
  }
}

.wallet {
  display: flex;
  flex-direction: column;

  .balance {
    text-transform: uppercase;
    letter-spacing: 1px;
    font-weight: 400;
    font-size: .9em;
    display: inline-flex;
    margin-top: 10px;
    padding: 0.75rem;
    background-color: #f4f6f8;
  }
}

.transfer {
  display: flex;
  flex-direction: column;
}

Create a server.js file and paste the code below;

import axios from "axios";

const server = axios.create({
  baseURL: "http://localhost:3042",
});

export default server;

This creates an Axios instance that uses http://localhost:3042 as the base URL, this is the URL our server will be listening on. The Axios instance is exported to be used by other services.

Now we would need to create a very important function. Here we would use cryptographic functions 😁. Specifically, the keccak256 hash function, which would be used to hash our transaction messages.

Create a `services.js` file, and paste the code below;

import { keccak256 } from "ethereum-cryptography/keccak"
import { utf8ToBytes, toHex } from "ethereum-cryptography/utils"

export async function hashMessage(message){
    return toHex(keccak256(utf8ToBytes(message)))
}

Now let's talk about what is happening here.

We have created a hashMessage function, which accepts a message (string) and returns a hash (string).

Keccak256, the function is imported from the Ethereum cryptography package installed earlier, it accepts bytes (Uint8Array) and returns a hash of fixed-sized output. The return type of this hash is a byte. The utf8ToBytes converts the message passed to the function to bytes which are then passed to the keccak256 function.

Finally, the function, toHex, takes the returned bytes from the keccak256 function and converts it to a hexadecimal string, which is then returned by the hashMessage function.

Its purpose is to encrypt a transaction message which would then be sent to the user to sign with their private key 🔑.

Creating the Wallet Component

The functionality of this component is to allow the user to input any address and see its balance. Create a Wallet.jsx file in the src directory, then copy and paste the code below into the file.

import server from "./server";

function Wallet({ address, setAddress, balance, setBalance }) {
  async function onChange(evt) {
    const address = evt.target.value;
    setAddress(address);
    if (address) {
      const {
        data: { balance },
      } = await server.get(`balance/${address}`);
      setBalance(balance);
    } else {
      setBalance(0);
    }
  }

  return (
    <div className="container wallet">
      <h1>Your Wallet</h1>

      <label>
        Wallet Address
        <input placeholder="Type an address, for example: 0x1" value={address} onChange={onChange}></input>
      </label>

      <div className="balance">Balance: {balance}</div>
    </div>
  );
}

export default Wallet;

Every time the input changes it will invoke the onChange event handler which will request the address balance from the server. If successful it will update the balance state using the setBalance action. The Balance and Address states are created from the App.jsx App component because their states will be used by the Transfer component too.

Transfer Component

This component will be used by the user to send any amount to any recipient. Create a Transfer.jsx file in the src directory, then copy and paste the code below into the file.

import { useState } from "react";
import server from "./server";
import { hashMessage } from "./services"

function Transfer({ address, setBalance }) {
  const [sendAmount, setSendAmount] = useState("");
  const [recipient, setRecipient] = useState("");

  const setValue = (setter) => (evt) => setter(evt.target.value);

  async function getSignature(evt){
    evt.preventDefault();

    try {
      let data = {
        recipient,
        amount: parseInt(sendAmount)
      }
      let msgHex = await hashMessage(JSON.stringify(data))
      let signature = prompt(`Sign message (${msgHex}) and provide signature:`)
      if (signature === null){
        alert("You did not provided a signature")
        return
      }
      await transfer(signature)
    } catch (ex) {
      alert(ex.response.data.message);
    }

  }

  async function transfer(signature) {
    const {
      data: { balance },
    } = await server.post(`send`, {
      sender: address,
      amount: parseInt(sendAmount),
      recipient,
      signature,
    });
    setBalance(balance);
    alert("Funds transferred successfully!")
  }

  return (
    <form className="container transfer" onSubmit={getSignature}>
      <h1>Send Transaction</h1>

      <label>
        Send Amount
        <input
          placeholder="1, 2, 3..."
          value={sendAmount}
          onChange={setValue(setSendAmount)}
        ></input>
      </label>

      <label>
        Recipient
        <input
          placeholder="Type an address, for example: 0x2"
          value={recipient}
          onChange={setValue(setRecipient)}
        ></input>
      </label>

      <input type="submit" className="button" value="Transfer" />
    </form>
  );
}

export default Transfer;

This is a lot, so let's break it down.

...
function Transfer({ address, setBalance })
...

The transfer component accepts two values;

• address - state object which is the sender's (user's) address

• setBalance - state action which will be used to update the user's balance

const [sendAmount, setSendAmount] = useState("");
const [recipient, setRecipient] = useState("");

These states are used to set the amount to be sent and the recipient's address.

...
const setValue = (setter) => (evt) => setter(evt.target.value);
...

This function takes in a callable, setter as an argument and returns an event handler. The function of the event handler is to pass the target value of the event object as an argument to setter.

It's used here;

...
<label>
  Send Amount
  <input
    placeholder="1, 2, 3..."
    value={sendAmount}
    onChange={setValue(setSendAmount)}
  ></input>
</label>

<label>
  Recipient
  <input
    placeholder="Type an address, for example: 0x2"
    value={recipient}
    onChange={setValue(setRecipient)}
  ></input>
</label>
...

The function, setValue , is used for the onChange events of both inputs above. Which updates the states of both amount and recipient with the values of their respective inputs.

Now, the function, getSignature , will be demystified. The goal here is that when the user enters a recipient address, the amount to be sent, and submits the form. A transaction object will be created which will contain the amount and recipient address, this transaction will then be displayed to the user so that they can sign it with their private key 🔑. Then the signature will be passed to the transfer function.

...
async function getSignature(evt){
  evt.preventDefault();

  try {
    let data = {
      recipient,
      amount: parseInt(sendAmount)
    }
    let msgHex = await hashMessage(JSON.stringify(data))
    let signature = prompt(`Sign message (${msgHex}) and provide signature:`)
    if (signature === null){
      alert("You did not provided a signature")
      return
    }
    await transfer(signature)
  } catch (ex) {
    alert(ex.response.data.message);
  }
}
...

The transfer function;

...
async function transfer(signature) {
  const {
    data: { balance },
  } = await server.post(`send`, {
    sender: address,
    amount: parseInt(sendAmount),
    recipient,
    signature,
  });
  setBalance(balance);
  alert("Funds transferred successfully!")
}
...

This sends the signature and the transaction data to the server. If the signature is valid for the transaction data, then the money will be transferred to the recipient's address.

That's all for the client-side application.

Server-Side NodeJS Application

The backend will be an express application. It's going to have two simple handlers;

• GET /balance/:address - to get an address balance

• POST /send - to transfer funds from one address to another.

Also, we would need to create some useful scripts for generating new random wallets and signing transactions. The latter is meant to help users sign transaction messages with their private key 🔑.

Firstly, let's set up the server directory. If not already created, create a new directory, server , in the root directory. Move into the directory, and run npm init -y to initialize the default package.json file.

Copy and paste the content below into the package.json file.

{
  "name": "server",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "generate": "node ./scripts/gen.js",
    "sign": "node ./scripts/signer.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "cors": "^2.8.5",
    "ethereum-cryptography": "^1.1.2",
    "express": "^4.18.1",
    "yargs": "^17.6.2"
  }
}

Then run npm install to install dependencies.

Before creating the scripts for generate and sign, we will be creating services.js first to define shared functions.

Services

Create a new file service.js in the server directory. Copy and paste the code below into it.

const secp = require("ethereum-cryptography/secp256k1");
const { keccak256 } = require("ethereum-cryptography/keccak")
const { utf8ToBytes, toHex } = require("ethereum-cryptography/utils")


function extractPublicKey(fullKey){
    let kec = keccak256(fullKey.slice(1, fullKey.length));
    return toHex(kec.slice(kec.length - 20, kec.length))
}


function verifySignature(sig, msg, pubKey){
    const msgHash = keccak256(utf8ToBytes(msg));
    let actualSignature = sig.slice(0, sig.length - 1)
    let recoveryBit = parseInt(sig[sig.length - 1])
    const sigPubKey = secp.recoverPublicKey(msgHash, actualSignature, recoveryBit);
    const mainKey = extractPublicKey(sigPubKey);
    return mainKey == pubKey
}

module.exports = {
    verifySignature,
    extractPublicKey
}

The function, extractPublicKey, accepts bytes array as an argument, this is the byte format of a full public key. Then hashes it with keccak256 and returns the hexadecimal string of the last 20 bytes of the hash. Ethereum docs explains why it's required. It's to make it shorter 😉.

The last function, verifySignature, accepts a signature, a transaction message, and the sender's public key. All this data is required to verify that the transaction message was indeed signed using the private key of the owner of the public key.

The function hashes the transaction message and used [recoverPublicKey](https://github.com/ethereum/js-ethereum-cryptography#secp256k1-curve:~:text=%3A%20boolean%0Afunction-,recoverPublicKey,-(msgHash%3A) to get the public key of the signer. After extracting the short format using extractPublicKey, it compares it will the public key, pubKey, passed to the function.

Script - generate: Generating new random public and private key pairs

const secp = require("ethereum-cryptography/secp256k1")
const { toHex } = require('ethereum-cryptography/utils')
const { extractPublicKey } = require('../services')


let privateKey = secp.utils.randomPrivateKey();
let pubKey = secp.getPublicKey(privateKey)

pubKey = extractPublicKey(pubKey)

console.log("Private key:", toHex(privateKey))
console.log("Public key:", pubKey)

A new private key is generated using the randomPrivatekey function. The public key is also extracted from the private key. The mathematical property that made this possible is so amazing. The private key can never be known from the public key.

Script usage example:

npm run generate

Console output:

Script - sign: Signing transaction messages

const yargs = require('yargs/yargs')
const { hideBin } = require('yargs/helpers')
const secp = require('ethereum-cryptography/secp256k1')
const { toHex } = require("ethereum-cryptography/utils")


let args = yargs(hideBin(process.argv))
    .option('private_key', {
        alias: 'p',
        type: 'string',
        description: 'Your Private Key',
        demandOption: true
    })
    .option('data', {
        alias: 'd',
        type: 'string',
        description: 'Payload to sign',
        demandOption: true
    })
    .parse()


let privKey = args.private_key
let msgHash = args.data

secp.sign(secp.utils.hexToBytes(msgHash), privKey, { recovered: true }).then(data => {
    const [signature, recovery_bit] = data
    let sig = toHex(signature);
    console.log("Your Signature:", sig)
    console.log("Your Recovery Bit:", recovery_bit)
    let fullSig = sig + recovery_bit.toString()
    console.log("Copy and paste this as the full signature, this has the recovery bit attached to the end:\n", fullSig)
})

This is more of a CLI tool. It accepts the private key and message hash as command-line arguments. The sign function from the module, ethereum-cryptography/secp256k1 , is used to sign the message hash, msgHash. The result of the signature and recovery_bit received from the data are later concatenated to form a single string, signature, which is logged to the console.

The signature is expected in the component, Transfer , in the client-side application.

So the user can copy it from the console and paste it into the prompt.

Script usage example

npm run sign -- -p 3ebefedbd43cbd88f0504acd101df139ddce0656da699b8350c1db9eaf193484 -d 3ebefedbd43cbd88f0504acd101df139ddce0656da699b8350c1db9eaf178970

Console output:

Now create the index.js file, which will contain our API routes and handlers, in the src directory.

Follow the steps below and paste the codes gradually.

const express = require("express");
const app = express();
const cors = require("cors");
const port = 3042;
const { verifySignature } = require("./services")


app.use(cors());
app.use(express.json());

const balances = {
  "KEY_A": 100,
  "KEY_B": 50,
  "KEY_C": 75,
  // KEY_N: Any amount
};

The object, balances , is currently acting as the database. The keys of the object will be public keys of different wallets, and the values will be their respective balance.

New public and private key pairs can be generated using the generate script. After creating any amount of key pairs, update the object, balances, with the public keys. Make sure to save their respective private keys too, so they can be used later for signing transaction messages.

GET - /balance/:address

app.get("/balance/:address", (req, res) => {
  const { address } = req.params;
  const balance = balances[address] || 0;
  res.send({ balance });
});

This is the route used to get an address balance.

POST - /send

app.post("/send", (req, res) => {
  const { sender, recipient, amount, signature } = req.body;

  const msg = JSON.stringify({
    recipient,
    amount
  })
  let isValid = verifySignature(signature, msg, sender);
  if (isValid === false){
    res.status(400).send({ message: "Invalid Signature!" })
    return
  }

  setInitialBalance(sender);
  setInitialBalance(recipient);

  if (balances[sender] < amount) {
    res.status(400).send({ message: "Not enough funds!" });
  } else {
    balances[sender] -= amount;
    balances[recipient] += amount;
    res.send({ balance: balances[sender] });
  }
});

This route verifies the amount and recipient, against the signature and public key of the sender. If the signature is valid, it tries to debit the sender, if this is successful it will update the recipient balance.

Lastly,

app.listen(port, () => {
  console.log(`Listening on port ${port}!`);
});

function setInitialBalance(address) {
  if (!balances[address]) {
    balances[address] = 0;
  }
}

Start the server to listen on any given port.

If the port is changed, make sure it's also updated in the client application.

The function, setInitialBalance, checks if an address exists in the database. If the address does not exist, it adds the address to the database with a balance of zero. This is a very nice method, as we don't need to manually add new users to our database.

Project codes are Github

Resources

Cryptographic tools for Ethereum - Ethereum Cryptography

Node CLI tool - Yargs

Play with Cryptographic hashes - SHA256 Online tool

Public Key Cryptography - RSA Algorithm & Elliptic Digital Curves (ECDSA)

Cryptography in HTTPS - Diffie Hellman Key Exchange & TLS handshake for HTTPS

Conclusion

The idea presented in this article can also be used in several fields of technology to maintain data integrity when data is transferred between several components through a network.

One flaw of the server built today is that it's a single node, which makes the database a centralized instance. This is where a blockchain ledger would be useful because it's managed by more nodes which are all bound by rules (consensus) making the whole system decentralized. An account with a public key, TEST, on node A, would have the same balance on node B.

Building applications on the blockchain will be covered later in new articles, subscribe to the newsletter to receive notifications when new articles drop.

Follow me on Twitter netrobeweb.