Eukarya Inc.

Company

Key Value
Company Eukarya Inc.
Employee 50+
Founded 2017
Web Site https://eukarya.io/
Description Eukarya develops Re:Earth (an open-source WebGIS platform), which has been adopted as the base system for a project led by the Japanese Ministry of Land
Location Tokyo, Japan

Team

Key Value
Title Software Engineer (Backend)
Mission My mission was to develop a workflow engine and microservices for Re:Earth Flow, an open-source ETL tool for GIS data. This platform has been adopted as the base system for a project led by the Japanese Ministry of Land.
Task
  • I developed new features for the platform.
Term October 1, 2024 - current
Team Size 10
Type Freelance

Projects

Key Value
Summary EU1. I developed a processor that compiles user scripts into WebAssembly files and executes them efficiently on a WASM runtime.
Problem
  • Need for User-Defined Functions: Users required the ability to integrate their own custom scripts (Python, Go, Rust, etc.) into the workflow engine to extend its functionality.
  • Lack of Extensibility for Custom Scripts: The existing engine lacked a mechanism to easily incorporate and execute these user-provided scripts.
Mission Enable Custom Script Execution via WebAssembly: My mission was to extend the workflow engine to securely execute user-defined scripts by compiling them to WebAssembly (WASM).
Action
  • Developed the WASM Extension:
    • WASM Compilation Process: I designed and implemented a compilation process that transformed user scripts into .wasm files.
    • WASM Runtime Integration: I integrated a WebAssembly runtime into the engine to execute the compiled .wasm scripts.
Challenge Ensuring Secure Execution Environment: A key challenge was to provide enough flexibility for users' custom scripts while strictly controlling the execution environment to prevent unintended operations and potential security risks.
Overcome Secured Wasm Execution with a Well-Defined API: I designed a well-defined API for the Wasm guest modules. This API strictly controlled how data (like variables) was passed and how host functions could be called, ensuring user scripts operated only within a secure and appropriate scope.
Result Significantly Enhanced System Extensibility: This feature greatly enhanced the workflow engine's extensibility, empowering users to efficiently customize their workflows with their preferred scripting languages.
Skill Feature Development
Key Value
Summary EU2. I developed a real-time logging system for the workflow engine, leveraging storage and cache efficiently.
Problem
  • Demand for Real-Time Workflow Monitoring: Users required immediate visibility into the execution status of their workflows.
  • Absence of Effective Logging System: The existing system lacked a robust mechanism for real-time logging and retrieval of workflow status information.
Mission Implement Real-Time Workflow Logging: My objective was to develop a real-time logging system enabling users to monitor workflow execution status directly from the user interface.
Action
  • Developed the Real-Time Logging Feature:
    • Integrated Pub/Sub for Log Publishing: I modified the workflow engine to publish log events to a Pub/Sub topic.
    • Developed Log Subscription Service: I created a subscriber service to consume logs from Pub/Sub and persist them to a data store.
    • Exposed Logs via API: I added an API endpoint (resolver) to enable the UI to fetch and display logs from the data store.
Challenge Meeting Real-Time Performance Demands: The primary challenge was to ensure the logging system could handle a high volume of logs and provide real-time updates to users without impacting overall system performance.
Overcome
  • Utilized Optimized Datastore Strategy:
    • Implemented Redis for Log Caching: I utilized Redis as a caching layer to ensure fast retrieval.
    • Utilized GCS for Archival Storage: Google Cloud Storage (GCS) was used for cost-effective, long-term storage of historical logs.
Result Achieved Real-Time Visibility and Enhanced Monitoring: The system successfully provided users with real-time visibility into worker execution status, significantly improving monitoring efficiency and troubleshooting capabilities.
Skill Performance Optimization

Technology

Value Tag
Go Backend
GraphQL Backend
MongoDB Backend
Redis Backend
Rust Backend
WebAssembly Backend
Google Cloud Infrastructure
Terraform Infrastructure