In software engineering, a software component is a modular, independent, and reusable unit of software that encapsulates specific functionality, with well-defined interfaces for interaction with other components. Components simplify development by allowing systems to be assembled like building blocks, encouraging reusability, maintainability, and scalability.
There are many real-world products and projects that are built using software components and component-based software engineering, such as Netflix's microservice-based and composable architecture, and Shopify's reusable React components in its Polaris design system. However, these components can only work in a company's specific domain and aren't interchangeable, i.e., Netflix cannot use Shopify-developed components and vice versa.
If software components made by different companies are interchangeable, it means their collaboration is through a consistant mechanism, including using a pre-agreed static data model for any underlying data exchange. This is difficult because the data model would ahere to each company's functional and business requirements, which are often evolving over time.
So instead of interchanging plug-n-play software modules, we often require extra effort to "integrate" two independently developed software, for their collaboration and exchanging data. For integrating larger scale applications, it typically requires building and maintaining dedicated middleware to bridge incompatible data models, but such an approach is impractical at the software components' granularity.
Indeed, for implementing truely interchangeable software components, we need a generic and consistent and static way for the components to collabrate, specifically, we need a generic plug-n-play component date-exhanging inteface, for all the current and future components, and such an interface cannot have a restricting data model because every component's data model is dynamically depend on its business and functional requirements and may (inevitably) change over time.
Q: What do Lego blocks have in common?
A: They all use the same type of joints for connecting to the other blocks.
From a product design perspective, the key to Lego's remarkable success is the standardized joint that appears in every Lego block. It's a feature that sets the foundation of the "modular architecture" of the entire Lego toy system, a feature that allows people to build arbitrarily expandable models without incurring exponential complexity. Lego joints allow any two Lego pieces to be uniformly locking and connecting together, regardless of how different their other aspects are. Inspired by this, we come to realise that, in a component-based modular software architecture, the key to control complexcity while more and more components are connected, is to simplify and standardize the way how components are connected - as in Lego, we need to develop a generic "component joint", or, in software development terms, a common component connection and data-exchanging interface.
Another analogy for a component-based software architecture is the breadboard, which is a tool used for wiring electrical components and developing prototype electronic circuits. Connecting electronic components and ICs' conductive metal wires and pins to a breadboard allows building a circuit, while these independent components transmit and exchange electrical signals through the connections provided by the board. The electronic components, such as transistors and ICs, are interchangeable as long as they have pins that are compatible with standard-sized plugs (holes) on the breadboard, and the breadboard's function is simply connecting the components and directing electrical signals according to the circuit. In this analogy, the component interface we seek to build is like the pin holes on a breadboard, and the data between the software components is like the electrical signals of a breadboard circuit.
Snappable is a library for such generic component interfacing that allows any compatible components to connect and exchange data between each other. Simply, it defines and implements the "joint" that connects software components, and just like the Lego joint, by allowing simple and unified connection and data exchange, it is the key for a component-based software construction framework.
Snappable implements such a "joint" as a unified, universal data transport layer API, through which components can send and receive unified data containers, and these containers can accommodate arbitrarily complex data. The data transport layer keeps the components connected regardless of how different or incompatible their internal data models are - meaning they can freely evolve their internal logic and data model1 without breaking the connection.
When assembling a modular app, Snappable allows compatible software components to easily exchange data between each other using the provided interface without having to be committed to a fixed data model.
Implementation-wise, the Snappable interface is "static" and maintenance-free, and it makes connecting and using compatible software components a close experience like assembling Lego models or building electric circuits using a breadboard.
From the application builder's perspective, these components are interchangeable because the interface they use for communication is generic and universal - just like you can change a household lightbulb as long as it fits into the standard socket. Thanks to the Snappable interface, components can be freely disconnected ("detached") from and reconnected ("re-attached") to an app, this means that potentially components made by different vendors are interchangeable.
So essentially, Snappable is a framework of data connections with sockets for connecting exchangeable software components - think of it as a wired house with sockets where you can plug in lightbulbs and electrical appliances.
Let's take a look at a working demo to visualize using the exchangeable components proposed above. This video below shows assembling an ETL app using pre-made Sanppable components which are avaiable from this repo.
This second clip explains how these components are configured indiviually and work collabourtively but are also interchangeable -
The program doing the demos is called SnapFusion, it contains a "runtime" environment that conforms the Sanppable API for providing the joining linkage connecting the components and implementing the data communication for the components data exchange needs2. From the fast development of modular apps perpsective, it may not appear to be anything exciting, but the true value of the Snappable API is that it defines "Snappable components" and the corresponding runtimes, based on which companies and developpers can make compatible components and runtimes that are interchangeable or be used together. In other words, you don't have to commit to use Foldda's SnapFusion runtime to take advantage the available Snappable components, and any Snappable-compatible components can be used with any Snappable-compatible runtimes3, or to be used in any general app as extension funtional modules (via the Snappable API).
Application builder - taking advantage of all available market components from all companies - loosely-coupled modular architecture leads to faster app development cycle, component interchangeablility leads to more choices, meaning portentially lower cost and higher quality thanks to competition.
Component developer - developing components that can be greatly re-used and easier to maintained, for bigger market.
Integration builder - connecting applications through using standard component-interface or by implementing lightweight "adaptor" components to bridge incompatible data models, rather than using high-cost middleware systems.
So how does Snappable work internally to deliver these benefits?
Snappable conceptually separates the components data exchange into two layers: the bottom layer, called the "data transport layer", is responsible for transferring the data content from one component to the other; and the top layer, called the "application layer", is responsible for interpretating the data content in the context of the application i.e., the context of the two components' interaction and collabration. The Snappable library only implements the data transport layer and the application layer is left to be implemented in the components.
Such a logical separation is the key to Snappable's software-component architecture, as the data transport layer is "the Lego joint" equivlant for component interfacing. Because the Snappable library is now only responsible for connecting and passing data from and to the components, and not for translating or interpretating the data, the generic interface it provides can be "schema-neutral" (i.e., being immute from the application's data model changes) and allow any component to connect and transfer data. The schema-neutral data transport layer is like the metal wiring from the breadboard connecting the electrical components, it is simply the path for the electrical signals passing thru, regardless of what the waveform and voltage the signals are.
And, by leaving the responsibility of describing or interpretating the data to the application layer which resides in the components, it elimites logical dependency to the physical connection and allows greater flexibility and lower costs in maintaining the components and the application.
In Snappable' schema-neutral data transport layer, it uses a universal data container class from the Charian data serialization API for moving the data. The container class, called Rda, has a recursive, multidimensional array structure which is also dynamically expandable. It provides a practically unlimited storage space that effectively can accommodate any arbitrary structured data.
The Snappable' schema-neutral data transport operates like the Post Office where everything are packaged inside carton boxes. In this analogy, the Snappable component interface is like the office counter where the components are the "customers" who send and receive their data via the "counter" using the provided Rda container "boxes". When components require exchanging data, they would be exchanging Rda containers through via the data transport layer (the "post office") provided by the Snappable API. Later, a component must unpack the Rda container to retrieve the stored data for consumption2 and the unpacking and consuming data operations are the application layer inside compoments themselves.
So essentially, Snappable is an API for components to freely exchage data, and it does so by defines a schema-neutral data transport layer where data are inside the unified Rda containers. The "container packing and unpacking" processes which are related to application-specific data models are responsible by the sending and the receiving components, in their application-specific implementation. By separating the two layers, data transport can becoming simple and unified and not being affected by the components' data model changes - we have now a Lego joint for joining components.
Leveraging the RDA universal data container, the Snappable API defines how a component can become a "snappable component" by implementing the ISnappable interface (explained below), and an environment where snappable components can use to send and receive data utilizing the underlying universal data transport layer. Using the post office as the analogy, snappalbe components like a "customer" must have certain characters such as having a name/id and delivery address, and the API provides a set of methods, like a post office's counter, for these component "customers" to deposit and to collect data packages.
consisit of set of classes and functions that a would utilize and exchange data with the other snappable components. The way how a snappable component using the provided data transport layer is very straight-forward, and can be conceptually explained as below -
n software architecture consist of a set of standardized classes and functions that plays their roles in an interactive scenario of how Snappable components can exchange data. In such highly abstracted scenario, a Snappable component is provided with a work environment consists of an input data source, and an output data sink. The component runs in a process loop, pulling input data from the input data source, do "some processing" about it, and dispose any output to the output data sink, and both the input and output data are in the form of RDA.
An abstraction of a Snappable component.
Post office counter and staff that facilitate the data transport.
A standardized data storage for holding data.
Any system implementing the Snappable API can benefit from its component-based computing architecture. For example, from this repo, there is a component called "HL7Networkreceiver", which can listen on a network port for receiving incoming HL7 messages. If your application requires such a function, you can implement the ISnappableManager interface, providing a "joint" where the HL7Networkreceiver can plug into and to dispose received messages to the output data store provided by your app.
You can also be a component developper, for example you can develop a component (implementing the ISnappable) that can covert input HL7 messages to write the data to a specialized database, and other people can use your component to join to the HL7Networkreceiver to assemble an app that can receive HL7 messages from the network and write to the database.
In these cases, both the apps and the components can be independently developped, components can be made available "on the shelf", and be used and re-used by any customer. And indeed, because of the standardized component joints, people can truely benefit from a much extended software component "market", for example, there can be many types of HL7Networkreceiver to choose from, and you can quickly swap and test and find the most suitable for you - just like choosing a light bulb from a harware store.
This project is released under GPL‑3.0 for open‑source use.
If you want to use it in a proprietary or closed‑source product, or distribute it without GPL obligations, a commercial license is available.
Commercial licensing offers:
- Permission for closed‑source usage
- Legal clarity for enterprises
- Optional support & long‑term maintenance
One objective of this project is to overcome one of the major challenge of implementing ture cross-vendor software component collabration and sharing. We have demonstrated that using the novel RDA encoding and the Charian serialization API, a small code-base API can achieve rather effective and practical component-based computing.
It shall be noted that even we are confident and positive with our code in this repo, the most value of this project is prove cross-vendor software component sharing is practically achievable, and key to this is implementing a generic data-transport layer that that features effective loose coupling to the connected components.
Through using Snapple ourselves, and through our clients, we have seen promising outcome that encourges us to continue enbrace and support this project, and we hope the software development would agree with us by sharing and same vision and hopefully benefit from, or even contribute to, what are provided here in this repo.
Obviosuly just because two components can be joined together and exchange data, doesn't necessarily mean they will work together smoothly, just like you can randomly connect two Lego pieces but the result may not be a interesting model that you wanted. Components work together need to understand the data they are sending and receiving. In the above exmples, the data type is the well defined HL7 message, so components designed to work with HL7 data type will work automatically. This is understandable just like a light bulb from a hardware store may only work with sockets supplies 220v AC.
Just like we have hardware circuits that can convert 220V AC to 5V DC, for cross domain, cross application data integration and interfacing, unless the receiver can handle multiple data types at once (which is possible), it is commonly require data type to be converted. In this repo, we see an example of such convertor component that converts HL7 data into delimited CSV format, so it can be written to a tabular database table.
If we compare Snappable components to electrical components, such transistors or ICs, a Snappble Runtime is the software version of "breadboard", that provides the sockets and wirings to connect the components.
Through the framework API, components, even without prior knowledge of each other, can be connected and exchange data and interact with each other in an app. This lays the ground for , and by doing so, it brings many benefits, such as rapid app development, more reliable software and lower cost resulted from a high degree of software (component) re-use. In an analogy, it's much like the hardware world of using the bolts and nuts purchased from hardware stores for use in home projects.
To achieve such a goal, the Snappable API must define what a component must implement, including -
- For being functional, the component needs to a way to perform a specified data-processing task,
- For handling data-processing task's input and output, the component needs to have have a way to exchange data with the other Enflow components.
- For being a physical assembly (i.e. "portable"), the component needs to be referencible by an OS-level physical computer object such as a file or a folder,
In addition to these, the framework API also defines a runtime must implement to run an Enflow-component-based app, so it's vendor-neutral, meaning the apps' components can be aquiried from open markets, and components with the same functions made by different vendors are interchangiable. This YouTube video gives a visual demonstration of the intended outcome of the framework, where Enflow components are assembled into data-processing solutions that can be deployed and run in a standard-compliant runtime environment.
Below we explain how the API is designed to specify these constrains, so the components and runtimes can interact with each other, performing their intended functions, within these standardized constrains.
In Emflow API's design, the process of using a component is modeled as a factory worker at a product processing line: the abstracted worker is given an "input container" which contains a co
Truly generic software components also mean they are freely interchangeable, "generic" software components, we need a mechanism that allows the components to flexibly interface with each other to exchange data, without being bound by their different data models (tight-coupling).
The Snappable component-based computing framework promotes high-level modular software re-use, by specifying a simple and practical open standard of software components and solution runtime, so both a component and a runtime can be independly developped, and are interchangible.
Functional implementations of the Enflow Components and Runtime are available pipelines for integration and data automation. "Portable" means Enflow components and assembled applications can be used on any computer4 without setup or installation.
SnapFusion leverages Charian, a universal data serializer, for connected components to exchange arbitrarily complex data, effectively allowing any Enflow-compliant components, from any person or company, to join and work together without pre-setting, and being constrained by, a data model.
Just like a LEGO joint can connect arbitrarily shaped pieces to function as a more complex unit, Enflow works by providing a "universal joint" for coupling software components to become a bigger module or an app.
Another analogy is that Enflow works like a breadboard where electrical components connect to it and work together as they use universal standard pins for sending and receiving signals. Enflow components send and receive data using the simple channel and format defined by the API.
This repo contains files and resources for Enflow component development, including -
- The framework's component and runtime API, for anyone to make compatible components, with arbitrary features that can collaborate and interact with Enflow components made by any other vendors.
- Source code of many Enflow components for various modularized tasks and functions, which can be used as they are, or as spoilerplates for further customizing a component tailored to your specific requirement.
- An easy-to-use "Developer Kit" program with GUI allows debugging and tracing the execution of connected Enflow components in Visual Studio.
Component-based computing promises many attractive benefits with its modularized architecture and interoperability feature, including better reusability, scalability, maintainability, and flexibility. However, implementing such a system has proven challenging and very few have succeeded. One fundamental problem to be addressed when implementing component-based computing is how components are going to connect and exchange data - to achieve maximum interoperability, minimum assumptions should be made between components about each other's data model and behavior, on the other hand, there shall be a level of "agreement" (i.e. a protocol) between the connected components about what and how data are exchanged for an interactive transaction. Note any such agreement being established would inevitably set a boundary to components about what can and cannot be done, so one of the design challenges for a true interoperable component-based computing system is to have the data exchange protocol as flexible as possible while maintaining operatable.
Einstein once said: Everything should be made as simple as possible, but not simpler.
One example of such an agreement is the REST API where schema
The ultimate goal of Enflow Component API is to become the base of an open-sourced software component marketplace, where free and premium components from different vendors are made available for people to assemble apps without programming. Not only component-based software development is much more productive and easier to maintain as you see in the demo, but a market of software components also has great economic value because it encourages a very high degree of software reuse. Theoretically, when a new component is developed and added to the market's collection, the number of possible apps from these components would multiply and grow exponentially, and, unlike using hardware electronic components, software components can be easily copied and reused in an app without much effort or additional cost.
In the past, despite these attractive benefits, one thing that stopped realizing component-based computing was how to define "a component's boundary" so it could co-exist and collaborate with the other components in an app. We need a standard interface that allows software components to freely and meaningfully exchange data.
Foldda Enflow is an attempt to solve the above problem, that is, it defines and implements such a "universal interface", for software components to exchange data while working together - even if the components are from different vendors, or developed at various times, and have little or no pre-established knowledge of each other4.
In an analogy, Enflow Components works much like "the breadboard for software" which essentially is a framework for simple and practical component-based computing. That is, like a breadboard defining the intended electric signals between electrical components and how they are connected, the API defines a generic data package standard for exchanging between software components and defines the interface of how each component can be connected and collaboratively function together in a runtime environment like Enflow. Think of a defining "universal plug" for software components that works like the pins and pin-holes in a physical breadboard project. Also, for a software component operatable like a physical electrical component, it has to be data-model-neutral, meaning the data exchange cannot be bound to a specific data model controlled by a vendor - think "the pin" and "pin-hole" for the breadboard have to be neutral and generic.
Based on the "pins" and "plugs" defined by the API. "Off-the-shelf" Enflow components can be made by any third party, giving you unrestricted choices of vendors for software components which you can use to create custom data processing and automation pipelines. And of course, you can also create components yourself without depending on a vendor.
An Enflow project (called a "solution") consists of a selection of components (called "handlers") that collectively work together to perform an application. Unlike the other modular software development frameworks, where software modules only exist in a proprietary IDE environment, Enflow components are packaged as file system folders, which can be physically carried in a USB, and be built into an app using plain Windows desktop operations such as drag-and-drop - i.e. without the need of an IDE. That is why building a Foldda app is more like a breadboard project except the outcome is a software application. This short video below is a demo of building and running an ETP pipeline with Foldda components.
As seen in the video, app-building with Enflow components does not require any vendor-specific tool, which means you can build or change an "Enflow app" from any bare Windows computer.
To achieve organic growth for the intended software component market, Enflow Framework must allow a user to modify a component, or to create new components, according to his/her specific requirement, rather than trying to provide a large number of components and try to satisfy all users' needs. So the API is designed to be (extremely) simple, flexible, and non-restrictive.
This repo hosts the open-sourced Foldda Automation API as well as the source code of many quality components developed by Foldda according to the API. These components can be used as they are, as you saw in the video, or serve as a boilerplate for you to customize or to start a brand-new component development, to suit your specific requirements. It is hoped these source codes will assist developers in understanding and developing their compatible software components.
The "Developer Kit" project included in this repo is a simple reference runtime. It is also designed to be used for the convenience of custom handler development as you can use it to debug your components' code by following a data processing flow across components.
In a Foldda app, each folder encapsulates a specific function of a data-processing step, the parent-children relationship of the stacked folders defines the data flow of the processing.
<>
When a Foldda app executes in a runtime, each module's logic (a specific data-process step) is turned into a process by the runtime, and the app's intended data-processing is performed sequentially as laid out by the folder's hierarchical structure.
<< foldda app execution with runtime >>
The framework is modeled as a factory processing line, where a worker (known as a "handler") takes items from an input bucket, processes them, and places the processed items (or other types of output) into an output bucket.
The Foldda "runtime" is the work environment for the workers, which includes providing the worker its input bucket, and output bucket, and, if applicable, passing the output from a worker to the next worker.
So in a Foldda handler, all it does is take data records from the provided input container, do the intended processing to these records, and then place the produced output to the provided output container. As defined by the framework, a Foldda handler would implement the IDataHandler interface -
public interface IDataHandler
{
/// Setting up the data-handler "worker" with its config, and its input and output storage
void Setup(IConfigProvider config, IDataStore inputStorage, IDataStore ouputStorage);
/// Typically runs a processing loop that processes the input records and saves the output records to the output storage.
Task ProcessData(CancellationToken cancellationToken);
}A Foldda runtime needs to address the problem of defining and implementing the interface between the components - which can be potentially independently developed and have no assumed knowledge of one other. And that is another key piece of tech from Foldda - the Charian object serialization API.
With Charian, Foldda runtime has this real power which is that it allows plug-n-play of third-party developed handlers that would work with existing handlers without having to recompile the app. It means you can have a handler built to your specific requirements while taking advantage of the existing prebuilt handlers, which means ultimate flexibility and control. And when a newly developed handler combines with the existing handlers, it multiplies the number of possible apps that can be built.
This allows Foldda Runtime to function as "the (software) breadboard", i.e. it powers up, and connects the input and the output of, the handler modules. More technically speaking, it navigates through a Foldda solution's folder hierarchy, executes the instructions in each module's folder, and provides data exchange between connected modules. An example of Foldda runtime is the Foldda Windows app.
Footnotes
-
While the data transport layer allows physically exchanging data during the components' collaborative interaction, there is still a "logical" application-layer typically implemented in the components themselves, where the data is interpreted and consumed. ↩
-
Foldda SnapFusion uses a Windows folder for wrapping and representing a software component where the files content inside the folder contains information, such has location pointers, about the actual software component being used. While it is visual and intuitive, you don't have to use Foldda runtime to enjoy the benefits of Snappable interchangeable software components. Just like you don't have to use a specific brand of breadboard for developping an electrical circuit, think Foldda is a specific brand of software-version of breadboard. ↩ ↩2
-
In fact, the Developper Kit project from this open-source repo is another fully functional runtime, although it is designed for developping and testing Snappable components. ↩