No matter your data needs, we know databases, storage, and reporting, on every scale.
On the scale of cell phones and small computer projects: SQLite,of course.
On the scale of local personal computers or office networks:
the classic PC based names you probably know: MYSQL, SQL Server
from Microsoft, Oracle - and for smaller projects, MS Access.
Some of these same names are to be found further up the infrastructure size scale, running on servers in data centers or the cloud – but for MS Access, which should stay on your local office network or server.
How large can we go here? Our favorite, MYSQL, can store up to 256 TB in a table.
( that "TB" stands for terabyte, or about 1024 gigabytes. This translates to about
200,000 5-minute songs; 310,000 pictures; or 500 hours worth of movies).
This is a lot of data, but there are other limits here, and we wouldn’t
suggest using MYSQL for this scale - such as the maximum file size
supported by the computer, as well as the efficiencies of the database.
In other words, for most needs, a classic database is fine. Indeed for hundreds of applications,
both on local networks and in the cloud, we’ve found MYSQL ( it’s free) or SQL Server or Oracle to be good.
But – if you have massive amounts of data – and if speed is vital.
we need to examine the next scale.
And this scale is defined by the kind of data you’re storing, as well as by where.
Back to the mainstream of databases at scale: There are other structural limits to classic databases; for example: MySQL can only use one CPU core per query, whereas newer creations, purpose built for large data, such as Spark, can use all cores on all cluster nodes.
Note the use of the term “cluster node”; the idea here is that of moving sections, “shards”, of the data to multiple machines, and then having a system of being able to query the entire cluster with a single query. In other words we now have an “operating system” built into the database, such that it can support this larger scale of multiple machines.
Our favorite system of this nature is Spark, created by Apache, who’s earlier incarnation Hadoop long held the title of most successful of the large database systems
Continuing our overview, all of the larger cloud systems ( Amazon, Google, Microsoft)
have proprietary large database system offerings in their cloud ecosystems.
Here you must ask yourself - do you need this scale? The advantages can be both the supported size, and automated scaling of processing power and storage. You’ll also benefit from offloading the infrastructure – until, perhaps, something goes wrong – and you find you have limited choices for data recovery. We can help with both the decision making and the later management of these systems.