#Day1 Query Engine and Storage Engine in a Database
A series of posts to understand databases better
In 2022, I got the chance to work on databases entirely, alhamdulillah.
1) I created my own database from scratch #flashdb (https://lnkd.in/gkP7fQ25)
2) I started work on a fast immutable database #immudb (https://lnkd.in/gcn6yry2)
To make database internals easy to understand for everyone, I am starting a small series #30daysofdatabases to understand how databases are constructed, helping engineers understand better the reasons behind performance bottlenecks in any database. Hopefully this inspired you to build one on your own.
Query Engine vs Storage Engine
The query engine and storage engine are two separate components of a database that perform different functions.
A query engine, also known as a query processor, is responsible for processing queries and returning the requested data to the user. It receives SQL or other query language statements (like PromQL or influxQL) from the application, and converts them into a series of instructions that the database server can understand. The query engine also performs tasks such as parsing and optimizing the query. ACID properties are usually implemented on the query engine of a database.
A storage engine is responsible for physically storing and retrieving the data within a database. It interacts with the file system or other storage subsystem to read and write data to disk or other storage media. The storage engine also manages the data structures and algorithms used to organize and retrieve the data, such as B-trees, LSM Trees, Merkle Trees, Append-only files, Paged Binary Trees, FD-trees, Lazy B-Trees or simple hash tables.
Notes:
- ACID transactions are usually implemented on the query engine of a database using a combination of techniques, such as two-phase commit, locking, and transaction logs, to ensure that the database remains in a consistent state, even in the presence of concurrent transactions, and that all changes made by a transaction are persisted to disk, so they can survive system failures.
- Query parsers basically take a string of text (in this case, a database query) and convert it into a data structure, such as an abstract syntax tree (AST), that can be more easily understood and processed by the database. Query parsers are usually pluggable in a well designed database. Here is an example of how we do it in #immudb https://lnkd.in/gDnw_AdS
In summary, the query engine is responsible for handling the communication between the user and the database server, interpreting and executing the query, while the storage engine is responsible for managing the physical storage of data, it uses different techniques and algorithms to store and retrieve the data efficiently. Together, these two layers form the foundation of a database management system, providing the functionality needed to store, retrieve, and manipulate data.
#database #sql #algorithms #systemdesign #arriqaaq #datastructures
Details on my work:
1) Building a database in Go from scratch: https://lnkd.in/gshJyeig
2) Working on a fast immutable database:
https://lnkd.in/gKMdJBTZ
